One of my favorite things about Python is the environmenting. I first learned virtualenv, then virtualenvwrapper, and now just pipenv, the current favored method to put a special location on top of the $PATH for a specific purpose.
PATH and venv/virtualenv explanation
Let’s talk a bit about how Python creates “environments.” When new to Python and to code isolation in general, some will compare a virtualenv to a virtual machine, or a docker container, or a remote server, however the actual truth is that nothing is actually isolated at all. A symlink to the Python interpreter you created the virtualenv with, as well as the location of any Python modules you have installed, are given to your current session as the FIRST place to look for executing or running anything.
We’ll get there but let’s talk briefly about the $PATH because even with professionals who have been working in technology for many years, I see this misunderstood, and it’s important. The $PATH variable is an environment variable (a variable that has value in your current session – so when you open up a new terminal) that is a _list of paths_, or a list of locations. Just as my address is 123 Main Street, and my friend’s address is 125 Main Street, you would have a reasonable guess as to where to find me. So in this analogy, let’s say that PATH="123 Main Street:125 Main Street". Your mac or linux machine works the same way, except the locations are directories like /usr/local/bin/ and /home/rachel/.local/bin and /sbin. The order of the directories is crucial – if you have an executable in the first path it finds, and in the third, your computer will STOP looking after it finds the first one!
So, the way that all Python environmenting, to my knowledge, works, is to change the $PATH variable to put the Python stuff, for the desired venv, at the very front of the $PATH. We would also say “on top of the $PATH,” because this is absolutely analogous to a stack. So with the Python interpreter you’ve specified, and the Python modules you’ve installed, and the $PATH changed to look for these FIRST, we have a venv! It’s important to understand that it’s really all on the same machine and with the same level of access – but we just artificially make the desired stuff MORE available.
Automate a bunch of virtualenvs at once
So, at my job, I’m working on improving onboarding. This is a really tall order with a lot of different vectors, but the technical aspects are all pretty exciting. First up, for whatever reason, I chose to work on the problem of how challenging it is to get a new engineer on our team going with a ton of different venvs. There’s a boatload of Python in our day-to-day, and laboriously going through and setting up the venv for 10+ places is a pain, and error-prone, and the way that we’ve been doing it is the old-school way, with virtualenvwrapper (a tool I’ve loved for a long time but which is starting to go stale with newer Pythons [and newer versions of macs]!), so weird things have begun to go wrong.
It’s not only time for an update to our methodology, it’s time to just abstract this problem away entirely. Folks on my team interact with Python but we don’t write much of it, but we must use venvs all day every day for various tasks because there’s just tons of tooling that’s been written in various versions of Python. So I wanted to write something that would just… DO IT. Something to take all the places where there should be a virtualenv, and make it based on spec.
There were a lot of places I went to research, I thought about Makefiles (really not the right tool), thought about Docker (make an image, not committed to the repo*, for every single one??? no), looked up some Python docs, and finally stumbled across this fabulous blog post, https://www.rootstrap.com/blog/how-to-manage-your-python-projects-with-pipenv-pyenv/ by Bruno Michetti, detailing how to make pyenv and pipenv play nicely together. Edit: And I assembled it all in bash, because that’s the system scripting language I’m most comfortable with. It’s not in Python, sorry! Mentally I consider this level of abstraction to be another level above Python, though this may be possible to bootstrap itself – I don’t really know.
Pyenv is a really lovely abstraction to download whaaaaatever version of Python you want, and switch amongst them very easily without having to know its location, whether or not it’s a symlink, yadda yadda.
Pipenv is a tool that bundles pip, which is the Python module install tool, with virtualenv! How sensible, given that the workflow previous to the existence of pipenv was to make a virtualenv, activate the virtualenv, and then pip install everything in requirements.txt. Pipenv condenses that with some nice bells and whistles besides.
The remaining challenge was getting it all into bash so the user could just run one thing, and organizing the script comprehensibly.
*not committed to the repo because this is not a task where I’m trying to convince fifteen teams to generate their module in a completely new way
The code
Typed this out by hand, I hope you like it, if you too have a ton of virtualenvs to install at once, or want to give this to a team to be able to use once you fill in the blanks of repo location & the repos themselves of Python modules using various Python interpreters! The only thing that’s not real are the repo names. It would be cool to get these dynamically – if you know how this could be done, based on a requirements file or something else, please holler in the comments!
#!/bin/bash
repo_location=$HOME/repos
# brew install pyenv & pipenv
brewing () {
brew install pyenv
brew install pipenv
brew update && brew upgrade pyenv && brew upgrade pipenv
}
# some tasks for pyenv to feel happy
pyenv_setup () {
if [[ $PIPENV_VENV_IN_PROJECT=1 ]]; then
echo "environment set up for pyenv already, probably"
else
echo 'if command -v pyenc 1>/dev/null 2>&1; then' >> ~/.zshrc
echo ' eval "$$(pyenv init -)"' >> ~/.zshrc
echo 'fi' >> ~/.zshrc
echo 'export PIPENV_VENV_IN_PROJECT=1' >> ~/.zshrc
source ~/.zshrc
fi
# if these already exist, the script will ask if you want to download them again
pyenv install 2.7.18
pyenv install 3.8.13
pyenv install 3.9.13
}
# now the juicy stuff! repo names are fake. eat your vegetables!
venv_creation () {
cd $repo_location
# py 3.8 envs:
three_eights="zucchini rutabaga cabbage"
for i in $(echo $three_eights); do
cd $i
pyenv local 3.8.13
pipenv install --python 3.8 -r requirements.txt
cd $repo_location
done
cd $repo_location # can you tell I've been scarred by bad dir management
# py 3.9 envs:
three_nines="cauliflower radicchio spinach"
for i in $(echo $three_nines); do
cd $i
pyenv local 3.9.13
pipenv install --python 3.9 -r requirements.txt
cd $repo_location
done
cd $repo_location
}
brewing
pyenv_setup
venv_creation
RKODE 