Modern Python Environmenting with Pyenv and Pipenv

One of my favorite things about Python is the environmenting. I first learned virtualenv, then virtualenvwrapper, and now just pipenv, the current favored method to put a special location on top of the $PATH for a specific purpose.

PATH and venv/virtualenv explanation

Let’s talk a bit about how Python creates “environments.” When new to Python and to code isolation in general, some will compare a virtualenv to a virtual machine, or a docker container, or a remote server, however the actual truth is that nothing is actually isolated at all. A symlink to the Python interpreter you created the virtualenv with, as well as the location of any Python modules you have installed, are given to your current session as the FIRST place to look for executing or running anything.

We’ll get there but let’s talk briefly about the $PATH because even with professionals who have been working in technology for many years, I see this misunderstood, and it’s important. The $PATH variable is an environment variable (a variable that has value in your current session – so when you open up a new terminal) that is a _list of paths_, or a list of locations. Just as my address is 123 Main Street, and my friend’s address is 125 Main Street, you would have a reasonable guess as to where to find me. So in this analogy, let’s say that PATH="123 Main Street:125 Main Street". Your mac or linux machine works the same way, except the locations are directories like /usr/local/bin/ and /home/rachel/.local/bin and /sbin. The order of the directories is crucial – if you have an executable in the first path it finds, and in the third, your computer will STOP looking after it finds the first one!

So, the way that all Python environmenting, to my knowledge, works, is to change the $PATH variable to put the Python stuff, for the desired venv, at the very front of the $PATH. We would also say “on top of the $PATH,” because this is absolutely analogous to a stack. So with the Python interpreter you’ve specified, and the Python modules you’ve installed, and the $PATH changed to look for these FIRST, we have a venv! It’s important to understand that it’s really all on the same machine and with the same level of access – but we just artificially make the desired stuff MORE available.

Automate a bunch of virtualenvs at once

So, at my job, I’m working on improving onboarding. This is a really tall order with a lot of different vectors, but the technical aspects are all pretty exciting. First up, for whatever reason, I chose to work on the problem of how challenging it is to get a new engineer on our team going with a ton of different venvs. There’s a boatload of Python in our day-to-day, and laboriously going through and setting up the venv for 10+ places is a pain, and error-prone, and the way that we’ve been doing it is the old-school way, with virtualenvwrapper (a tool I’ve loved for a long time but which is starting to go stale with newer Pythons [and newer versions of macs]!), so weird things have begun to go wrong.

It’s not only time for an update to our methodology, it’s time to just abstract this problem away entirely. Folks on my team interact with Python but we don’t write much of it, but we must use venvs all day every day for various tasks because there’s just tons of tooling that’s been written in various versions of Python. So I wanted to write something that would just… DO IT. Something to take all the places where there should be a virtualenv, and make it based on spec.

There were a lot of places I went to research, I thought about Makefiles (really not the right tool), thought about Docker (make an image, not committed to the repo*, for every single one??? no), looked up some Python docs, and finally stumbled across this fabulous blog post, https://www.rootstrap.com/blog/how-to-manage-your-python-projects-with-pipenv-pyenv/ by Bruno Michetti, detailing how to make pyenv and pipenv play nicely together. Edit: And I assembled it all in bash, because that’s the system scripting language I’m most comfortable with. It’s not in Python, sorry! Mentally I consider this level of abstraction to be another level above Python, though this may be possible to bootstrap itself – I don’t really know.

Pyenv is a really lovely abstraction to download whaaaaatever version of Python you want, and switch amongst them very easily without having to know its location, whether or not it’s a symlink, yadda yadda.

Pipenv is a tool that bundles pip, which is the Python module install tool, with virtualenv! How sensible, given that the workflow previous to the existence of pipenv was to make a virtualenv, activate the virtualenv, and then pip install everything in requirements.txt. Pipenv condenses that with some nice bells and whistles besides.

The remaining challenge was getting it all into bash so the user could just run one thing, and organizing the script comprehensibly.

*not committed to the repo because this is not a task where I’m trying to convince fifteen teams to generate their module in a completely new way

The code

Typed this out by hand, I hope you like it, if you too have a ton of virtualenvs to install at once, or want to give this to a team to be able to use once you fill in the blanks of repo location & the repos themselves of Python modules using various Python interpreters! The only thing that’s not real are the repo names. It would be cool to get these dynamically – if you know how this could be done, based on a requirements file or something else, please holler in the comments!

#!/bin/bash

repo_location=$HOME/repos

# brew install pyenv & pipenv
brewing () {
    brew install pyenv
    brew install pipenv
    brew update && brew upgrade pyenv && brew upgrade pipenv
}

# some tasks for pyenv to feel happy
pyenv_setup () {
    if [[ $PIPENV_VENV_IN_PROJECT=1 ]]; then
        echo "environment set up for pyenv already, probably"
    else
        echo 'if command -v pyenc 1>/dev/null 2>&1; then' >> ~/.zshrc
        echo '  eval "$$(pyenv init -)"' >> ~/.zshrc
        echo 'fi' >> ~/.zshrc
        echo 'export PIPENV_VENV_IN_PROJECT=1' >> ~/.zshrc
        source ~/.zshrc
    fi

    # if these already exist, the script will ask if you want to download them again
    pyenv install 2.7.18
    pyenv install 3.8.13
    pyenv install 3.9.13
}

# now the juicy stuff!  repo names are fake.  eat your vegetables!
venv_creation () {
    cd $repo_location

    # py 3.8 envs:
    three_eights="zucchini rutabaga cabbage"
    for i in $(echo $three_eights); do
        cd $i
        pyenv local 3.8.13
        pipenv install --python 3.8 -r requirements.txt
        cd $repo_location
    done
    cd $repo_location   # can you tell I've been scarred by bad dir management

    # py 3.9 envs:
    three_nines="cauliflower radicchio spinach"
    for i in $(echo $three_nines); do
        cd $i
        pyenv local 3.9.13
        pipenv install --python 3.9 -r requirements.txt
        cd $repo_location
    done
    cd $repo_location
}

brewing
pyenv_setup
venv_creation

EOY 2014

Whew. I thought 2013 was fast. 2014 was bananas. I think I’m a bit rusty from not writing as much as I ought to, so let’s just jump right in!

Grad School?

In February, I received a letter of acceptance from my absolute dream graduate school to teach high school mathematics. In what has become a theme, it was something I’d worked incredibly hard for over a long period of time to achieve, and I actually made it. I knew what I would do before I even got the letter, but that didn’t actually make it any easier – I turned them down, not only because it would have increased my debt by 130% from what is already a serious amount of money, and not only because I never would have made anywhere near enough to have repaid this ~$100k of loans, but for a host of other reasons as well. With a heavy heart, I emailed my amazing advisor and advocate at the school, a nearly two year relationship, to let her know I wouldn’t be enrolling.

So I threw myself into computer science study. With PyLadies, and my last quarter of school to complete my French degree and Math minor, I busted the proverbial it to get a job, which I did, literally the day I was done with my final FINAL exams, hopefully ever : )

Interne-towne

The job started as an internship with lots of hands-on pair coding with my boss, “how would you solve this problem,” “let’s refactor together,” along with some “hey would you reach out to this person to sponsor (x),” which was great! I got to use a way nicer computer (macbook O2) than I had (a lovely old giant brick of a PC laptop on which I installed Ubuntu 12.04), and I finally started using git at the command line. It became pretty quickly evident that we worked together fabulously, and we started to think about what kinds of projects we could do together, so we started working on new ideas, largely centered on education. It was, actually, an incredible collaboration.

We thought topics and people for Security in Python, Data Science in Python, Twisted, Django From the Actual Beginning, and a few more that I’m sure I’m missing – it was rather a fire-hose of ideas! My boss was the kind of person who had six good ideas before breakfast, and it was a fast-paced, sometimes stressful, REALLY productive space for six months.

While there, I learned git to a granular degree and now lead a monthly workshop on it and plan to lead/teach the PyLadies annual course on it as well, and while I didn’t improve my Python chops much, I learned a lot about how computers are really working, under the hood – well, under a relative hood, I got into no hardware, lol, not at all. The os and sys modules, jeez! I’ve been saying lately that it’s those two modules that turn Python from an expensive calculator into something really powerful.

Codecademy played a little role, too, as you can see if you search the javascript tag on here – while I don’t code in JS, it’s a fairly ubiquitous language & I’m glad to have some familiarity with how it handles different kinds of problems.

I also, god-willing, learned a bit of project & people management the hard way. I don’t ever want to do that again, hooray! It’s good to know, especially considering that women are often directed from engineering career paths to soft-skill positions – now I know what to push back against.

Tutorial Creation

We settled into a Python tutorial with a local $TOPIC_IN_PYTHON expert, and worked really hard on outlining, scoping, creation, refinement, refactoring, presentation, program executability, git monsters, project jupyter/pip/virtualenv & dependency concerns, and so many other logistical issues.

After we flew to the place to film it, the company we’d signed the contract offered a counter-contract, letting us off the hook, and decided not to publish it after all. While honestly heartbreaking, there was a serious amount that we all learned.

UNEMPLOYMENT OH NOES

While scrambling to find a new gig after my internship ended, I stumbled on a number of really awesome opportunities, and though I only was offered one – obviously I stopped looking once I got an offer – I met and now keep in touch with many of the folks I interviewed with, because while it didn’t work out, these are all really neat people at really cool companies, so that has been really validating.

I was strictly unemployed for all of two weeks before I got an offer from Puppet, where I’ve been since mid October, and totally in love with my job. 2014! WOW.

2015?

This year, I will be learning the ins and outs of system administration, I’ll become Puppet certified, I’ll learn Ruby, and I’ll be dipping my toe into web app development as well, all while being part of the neatest leadership team of ladies ever with PyLadies PDX. I think I’d like to get Red Hat certified, or close to it, as I’ve come into Puppet without any sys admin background, and that’s something I’d really, really like to be good at at the new spot. I continue to be amazed by how well I am treated in my new career and how much I can do for people, even while clawing at so much more – frustration at the not-knowing-enough is a constant underlying anxiety for me at the new job, and I suspect that will continue for some time, and which in fact is a good thing. Studying math, and trying to stay aware of what’s Going On in tech, is a pretty good primer for the huge amounts of Not Knowing involved in working in tech, for probably at least the first few years.

WHOO! Onward and upward, or as they say, “Up and to the right!”

Beachmeals

It’s been a month – the longest I’ve gone since I started this blog back in, hm, September 2013? About a month ago, my friends and I decided to rent a house for a weekend. There were nine of us, so I wrote a program to assign people randomly to each of the four meals we would be preparing that weekend – Saturday brunch, weekend snacks, Saturday dinner, and Sunday breakfast.

At first, and I knew this would not be my final draft, I made a loop to ask how many people would attend & another to ask all their names. Then, I made an empty list for each meal. Then I made a for loop to assign each person to a given meal, for the number of people attending. Each chunk of the code looked much like this:

appender = friendForIndex[-1]
brunch.append(appender)
print "adding guy above to brunch"
list.pop(friendForIndex)
newIndex = newIndex - 1
print "brunch folxxx: %s" % brunch

Later grawnkps (a very technical term, that) had an altered meal, so rather than brunch of course it would have breakfast or whatever.

It worked, which is fine, but you know me, I had to keep fiddling – 80-odd lines is too many for something that seems so much simpler than this! And since each grawnkp was essentially the same block of code over and over, well, obviously SOMETHING can be done about that!

So I set to re-writing. Another week or so of off-hours fiddling led to the final portion of the code as follows, a (sometimes) triply-nested conditional loop, that rather than needing one individual grawnkp for each meal, it iterates over a list of meals and picks the next one if the indexing value is over 0, and after each iteration it subtracts from the index within the bottom-most conditional, so it doesn’t iterate unnecessarily, which is what I was nervous about, but I did it right!! Check it out, I am quite proud. meals is a list of meals with brunch, snax, dinner, and breakfast as empty lists declared within. and shuffledFriends is a copy of the list of friends which has then been randomized:

k = len(shuffledFriends)
while k > 0:
	for j in meals:
		appender = shuffledFriends[-1]
		j.append(appender)
		list.pop(shuffledFriends)
		k = k - 1
		if k > 0:
			pass
		elif k <= 0:
			print "brunch fixers: %s \n" % brunch
			print "snax fixers: %s \n" % snax
			print 'dinner fixers: %s \n"; % dinner
			print "breakfast fixers: %s \n" % breakfast
			print "good job now make the food you dooks"
			exit(0)
		else:
			print "error"

ed: fixed tablature error with [ code language = “somelang!” ] codehere [ / code ]

So I am really excited! The next step for something like this could be to import a file with some of this information, but it works REALLY WELL for what I was trying to accomplish. Here’s the repo if you’re interested in how the whole thing fits together. The first version is the master branch in there so take a look if you’re so inclined, though like I said, it’s not nearly as flashy!

yahoo! & by way of an update I think I’m done with Learn Python the Hard Way. I have learned a lot & now am finally getting into some of my other projects.

Rinance

In trying to set up a proper website for myself, which I won’t link til it’s at least somewhat how I want it to look, I have been a little frustrated, so as a side project, I am trying to make myself an eventual budgeting program, based on my brother’s Brinance, and by based on, I don’t mean based on at all – I am going to write my own, ha ha! and then see how his differs from mine and in the process learn how to read a bit of perl. Exciting!

Here‘s what I’ve got so far.

I want to try to bang out the rest of LPTHW over my winter break, which has just begun. We shall see if that is doable! Then I want to move on to Learn Ruby The Hard Way, as Ruby is super-sexy right now.

brb trying to install django

wish me luck!

EDIT: so. this installation has been Really Hard. finally, having installed wsig, virtualenv, and virtualenvwrapper, I finally installed pip to handle python-related downloads, and while muddling yet again (really – for like the tenth time over the last couple days) through trying to just download django, my friend suggested, “well, why not just sudo pip install django?” HA! beautiful! so that worked, immediately, and now I have django on my linux box. woo-hoo!

now to USE it, to make a django-structured (probably wrong terminology) personal site, to showcase my ~tAlEnTs~! aaaahhhh