JOB TOWNE: JOB HARDER

I am looking for work! If you’ve been browsing my blog long, or not, you’ll know that I’m primarily a backend-focused Python developer with config management, virtualization, and documentarian bents. My peopling & coworking is LEGIT and I love mentorship and thinking about information transmission. I’m also interested in tech writing, if it’s for something good and chewy that I personally want to see more documentation on (read: everything complicated), and I’d consider a dip into devops/SRE too.

Some of my musts include the following:

  • An established team of at least several years. It seems like between 5-10 years is a very good sweet spot for the kind of growth I’m looking for.
  • You use the Agile development strategy, or something similarly modern. Sprints, clear work assignment and tracking, post-mortems.
  • You use safe, modern Git practices.
  • You have other women in the company.
  • You have onboarded people before.
  • You, as an organization, have made an attempt at writing internal documentation.
  • I am happy to work remotely, but I do not want to be your only remote worker.
  • I am happiER to work in Portland, and require the flexibility to work from home a few times per month, once onboarded.
  • A semi-dedicated resource of whom I can ask friendly questions for the first several months.

Some of my wants:

  • To not be the first woman engineer you hire. This has been very difficult to find.
  • To primarily use Python, with the flexibility to learn new languages.
  • To have the time granted to write great documentation along with the features and fixes I write for you.
  • To be part of a rich code-reviewing team, where everyone’s commits are reviewed, even architects’.

Leave a comment or email me at rkellyalso aat gmail and I’ll shoot you my resume. Let’s doo thiiis.

“Composting” Docker

Right now I’m fiddling with Docker and combing through their terrific, extensive docs. As I have a history of doing lately with this blog, I’ll talk about my own setup and the installation process, all the way to the Docker image I’ve created and messed with, and what’s next for my own knowledge.

New laptop

First, I recently purchased a machine from Free Geek so I’m going through the delightful process (really :D) of setting it up. When I volunteered at Free Geek (which I HIGHLY recommend, please ping me for details!!), the computer I received after building desktops for them for a while was 40G HDD, 512MB DDR2 RAM, and was capable of a wired-only connection to the internet. After a USB wifi antenna, the machine could do all the browsing I needed, which was all I did at the time. At the time, that kind of machine cost anywhere between $80-$130 if my memory serves me correctly. I believe this was about 2011.

Two days ago, I wanted a laptop that would be able to run VMs without too much trouble, and I wanted to spend $200 or less as I am unemployed. Five years after my volunteer stint, I got a machine with 8G ram DDR3, 250G SATA HDD, i5 dual-core processor, in a pretty giant old Dell Latitude E63340, for exactly $200. AMAZING. I’m astounded at how cheap it is. And what else has changed? It’s no longer Ubuntu that they ship the computers they clean, reassemble to spec, and sell or give away, it’s Linux Mint, specifically Edition 17.1. Thoughts so far are really just that it feels very Windows-y with its focus on the (not) Start menu, and you’ll get no complaints from me on this front. It’s Ubuntu with a coat of paint on top, and I’m not toooootally sure this is what I want to be running, but it definitely works and there is probably broadest reach of packages available for the end-user (not enterprise which is of course another red-colored ball of wax) so it will do, for now, and at some point I am sure I’ll explore other distros.

Docker

On to Docker! I’ve previously read up on Docker, and find its resource allocation methodology super interesting. I’ve used a lot of Vagrant/VMWare on my OSX box at my previous job, and while the mechanism to spin up/destroy boxes was pretty good, the amount of cruft left behind became frustrating at a work level – I had to clear out vast swaths of space with a machete on an irritatingly regular basis! Basically, Docker runs without a hypervisor, which in the spirit of keeping things in layperson’s terms, is a virtual machine management machine which is pretty heavy. That they’ve figured out a way around the heaviness of a hypervisor is a Big Deal. And as I understand it, it uses the resources of the host machine very intelligently. With the previous VM paradigm, you gave both disk and memory permanently (insomuch as a VM that you create & tear down relatively easily is permanent) to the machine, and the VM held onto that inaccessibly while in use. No such selfish tactics are employed by Docker and use of memory in particular is supposedly much more elastic.

Installation was just a few shell commands which worked with no fuss, I used this official site. The instructions are for Ubuntu 14.04 which Mint 17 is based on, and as I’ve covered above, all Ubuntu packages can be used on Linux Mint with little to no adaptation required.

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates
sudo apt-get install linux-image-extra-$(uname -r)
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' >> /etc/apt/sources.list.d/docker.list
sudo apt-get install docker-engine
sudo apt-get update

Then, the Docker engine can be started! First start the service (this may be different if your linux distro uses systemctl, like a newer version of Fedora or RHEL/CentOS) & then make sure their dummy example works.

sudo service docker start
sudo docker run hello-world

Ok! Now we’ve got Docker installed. If you encounter anything funny here during installation I’d love to hear about it, please leave a comment! I can try to help.

There are two primary ways for the new end-user to continue forward. One of them is writing a Dockerfile. I leave that for its own blog post, but hold me to it! I want to write a few basic Dockerfiles for my own reference & I imagine they’ll be of some use to others! The second way I’ll mention here is via DockerHub, which you don’t even directly need to interact with to use! To me it’s reminiscent of pulling images down from VagrantCloud, but again, you don’t even have to go there to grab things. You can feed a few different common distros into the docker run command and it will pop you right into a container of that OS! It’s still rather magical at this point, so I’m still learning more about it so it becomes a bit less magical😉

Getting into the box I wanted was as simple as plugging in the distro I wanted, Fedora, into the docker run command, like so:

sudo docker run -it fedora /bin/bash

What this led to for me, which I need to learn SO much more about, is an EXTREMELY spare version of Fedora that has very few executables I’m used to. Because it’s the latest version of Fedora, its package manager is dnf & not yum, but it still knows what you mean & permits installation “via” yum – but really it’s just aliased to /usr/bin/dnf, haha, which is fine.

Quick fun fact: dnf is an abbreviation of “Dandified Yellowdog Updater, Modified” – you can see the yum in there as the commonly used RHEL and RHEL-flavored linuxes’ package manager. To me, whenever I type dnf install -- it looks as though I’m typing “do not f&$%ing install”😀

So, next steps, make the container persistent – it isn’t yet, and play around with Dockerfiles, and just do a little more spying on the produced container itself & probably try to do some babby’s frist load testing things in there & spy on the container as a process without the box & all its processes within!

See you next time! Would love to hear from you. If I’ve missed the mark on anything with this pretty chewy piece of technology please let me know, or if there’s anything you’d like to see me cover leave a comment!

btw the title of this is a joke which was made during the Docker-fiddling open space I held at Open Source Bridge 2016🙂 if you were there, thanks for coming! Super fun discussion.

Postgres on Fedora

Note: This is a post from several months ago in the ol’ drafts bin and there’s a ton of information here, even though it is incomplete. I’m not running Fedora any more, but it’s possible this could help someone else, so I hit publish.

This shouldn’t be too long a post, but I’ve encountered something that does not really feel like it ought to be an edge case!

In trying to install PostgreSQL on Fedora 23 I ran into a few snags per the Other Linux Installation book published here using the download guide for RH-flavored Linuxes here.

Sidebar: In the Installation section of the introduction to Postgres (I know PG pretty well in the context of Puppet Enterprise, but I really want to expand that knowledge since I know a lot of people use & love it [/diatribe on why I’m doing this]) it says the following:

If you are installing PostgreSQL yourself, then refer to Chapter 15 for instructions on installation, and return to this guide when the installation is complete. Be sure to follow closely the section about setting up the appropriate environment variables.

That’s in section 1.1, on the Installation page, by the way. This might be why we can’t have nice things.

MOVING ALONG, the way which has worked for me to install Postgres is the following, thanks in much part to a) This Fedora Project doc and b) working knowledge of the su command.

$ sudo dnf install postgresql-server postgresql-contrib
 ... "are you sure you want to install y/N" ...

$ sudo systemctl enable postgresql
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql.service to /usr/lib/systemd/system/postgresql.service.

$ sudo postgresql-setup initdb
WARNING: using obsoleted argument syntax, try --help
WARNING: arguments transformed to: postgresql-setup --initdb --unit postgresql
 * Initializing database in '/var/lib/pgsql/data'
 * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log

$ sudo systemctl start postgresql

So, only after making the dummy initdb did the systemctl start command go through, but I noticed that there was still no pgsql executable, so I couldn’t actually use postgres yet. Finally, I catted the postgres log like so:

$ sudo cat /var/lib/pgsql/initdb_postgresql.log
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/bin/postgres -D /var/lib/pgsql/data
or
    /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start

which, up at the top, points out that that this needs to be run with the postgres user, and after some fiddling, knowing that the -D flag needs to be before the /var/lib/pgsql/data path as designation. Also seeing su crap itself a number of times was irritating. To get into the postgres user, it was necessary to provide a password! I hadn’t set one, so I tried a couple easy guesses & couldn’t figure it out and a (very quick) DDG didn’t yield anything either, so I snuck around it with sudo -u postgres psql, which only asked me for the superuser password – A-OK! But Then! It then complained that it didn’t have permissions to get into ~, but reasonably got me into the postgres user’s prompt: HERE, FINALLY, I was able to run

postgres=# /usr/bin/postgres -D /var/lib/pgsql/data

Though… now I’m noticing that anything I type in there doesn’t even throw an error. It even offers ‘Type “help” for help,’ and yet when I do, with and without quotes, there is no output & no result. And I still don’t have a pgsql executable.

Ok, but there IS a result from which postgres, which is the nicely predictable /usr/bin/postgres. One success – I have a universally executable postgres! So when I run it with no arguments, hoping for more information from my sleuthing, I get some!

 $ postgres
postgres does not know where to find the server configuration file.
You must specify the --config-file or -D invocation option or set the PGDATA environment variable.

So that’s where the -D is necessary. Cool. But we also definitely don’t have the PGDATA env var
postgres -D /var/lib/pgsql/data.

Wow! There’s more to do here, and I don’t have time right now, so have SOME information, yet incomplete!

Setting up my Fedora workstation

Please note that this post contains content not suitable for those who give no craps about the way a Linux box can be set up for an end-user. I do not blame you, those who give no craps.

So leaving Puppet after 18mo (on great terms! hi friends!) I find myself in need of my own real development machine. I begrudgingly find myself admiring macs after all, but after two minutes of looking at craigslist and finding $700 MBPs from five years ago, concluded that that Just Isn’t Going To Happen, much as I love iTerm2. I’ll buy myself a new computer soon but in the mean time a friend of mine had an Ubuntu 12.04 ThinkPad X220 she said I could borrow for a bit, and oh my god, I am in LOVE. This machine is great and zippy and POWERFUL. I might.. I might just buy a clone when I’ve got the bucks, even though you can pretty much only get them used at this point.

Rather than jump back into Ubuntu which is pretty familiar ground, I wanted something slightly different and my friend Amy has been extolling Fedora’s virtues for years. Further, at Puppet, we virtualized nearly all our testbeds in CentOS using the amazing, moooostly internal (but totally available!) Puppet Debug Kit created & maintained by my brilliant former coworker who is still doing phenomenal work over there. Ok, so I will definitely miss my buds there!

So because I spent about half my time on the job in CentOS & Fedora is the closest end-user version of that with a UI (sorry I’m not hardcore enough to only run a server for my dev box haha!), I grabbed the instructions & made a Fedora-specific bootable usb drive with their (prev linked) docs. After formatting the drive, writing the .iso to it, and plugging it in, I had to fiddle with the BIOS, which on the x220 was incredibly easy – first, on bootup it tells you EXACTLY how to get into BIOS, and it gives you the option to do a one-time boot via USB, rather than having to muck around with boot order! Fabulous!! Then with a bit of wiggling (had to get into a babby command line rq to tell it to choose the Linux0 option which kicked off the install, please, friends, do not ask me why) the installation went off without a hitch, with LITERALLY NONE HITCHES.

It was after rebooting that I started to learn how powerful this little machine really is. It’s fast, despite having 1/4 the memory of my old work MBP (though I really don’t know how that scales), and the trackpad uses all the gestures I’m used to from working on macs.

Then I set up my prompt, and without wanting to get toooo too deep into the oogly bits of bash formatting, I had to try and test and try and test and finally settled from:

export PS1='\[\e[0;36m\]rk\[\e[m\] \[\e[1;37m\@ \w \[\e[m\] \n $ '

which threw a non-ASCII character, and when I fiddled, lost the ability to shut off the bold white text, haha, to:

export PS1='\[\e[0;36m\]rk\[\e[m\] \[\e[1;37m\@ \w \e[m\] \n $ '

Huh. That’s only one [ different. Just bless ya, monospace blog draft.

Anyway, then I got ambitious. I wanted to see if I could run Spotify outside a webapp, because that makes it IMMEDIATELY less likely to be used and I rely pretty heavily on it, during and outside the workday. Using these set of instructions which state a requirement of RPM Fusion as installable here, I got going. These are for Fedora 20 & I’m on 23, but I knew I could get it going. I was so excited for this, I LOVE a new Linux system’s first sudo yum(or whatever) update, so I ran that & a few minutes later tried to get RPM Fusion itself installed with the following command:

su -c ‘yum localinstall –nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm’

But it griped at me about there being no localinstall user – it was griping because we’d told it to perform a command with a specified user with the su command, but it had received no user. Usually this should result in its just using root, so it’s close to the same as just using sudo in front of important things you run in the terminal, but my bash version 4.3.42 was having none of it. So I peeled out the su -c (the -c just means you’re passing it a command to execute immediately, then return to the normal user after execution, rather than switching wholly into the specified user). The issue I ran into thereafter was still localinstall, which my machine still couldn’t find. I made a few attempts at installing localinstall (so meta) but it escaped me. I found this Stack Overflow-ish post asking about basically the same difficulty I was having, and more or less someone says that yum install and yum localinstall accomplish the same thing and the only reason the other still exists is for backwards compatibility. So I changed localinstall to install, removed the su -c ' ', added a sudo since the yum install would want it, and BAM, RPM Fusion on Fedora 23!

sudo yum install http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Then, all there was to do was run the lil commands to actually get Spotify since RPM Fusion’s installed! The “dnf” of the Fedora package manager command cracks me up – sounds a lot like “do not f****ng” before “install blahblahpackage”, and I refuse to look up what it means because I laugh every time.

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-spotify.repo
dnf install spotify-client

And that’s all it actually took, which, haha, looking back at what I’ve written, I guess is slightly more complicated than “that’s all it took” might warrant.

Next I need to find a terminal I’m happier with! I seriously miss iTerm2 so if you have any Fedora-flavored terminal loves let me know in the comments. I need tabs, man. I need ’em.

EOY 2014

Whew. I thought 2013 was fast. 2014 was bananas. I think I’m a bit rusty from not writing as much as I ought to, so let’s just jump right in!

Grad School?

In February, I received a letter of acceptance from my absolute dream graduate school to teach high school mathematics. In what has become a theme, it was something I’d worked incredibly hard for over a long period of time to achieve, and I actually made it. I knew what I would do before I even got the letter, but that didn’t actually make it any easier – I turned them down, not only because it would have increased my debt by 130% from what is already a serious amount of money, and not only because I never would have made anywhere near enough to have repaid this ~$100k of loans, but for a host of other reasons as well. With a heavy heart, I emailed my amazing advisor and advocate at the school, a nearly two year relationship, to let her know I wouldn’t be enrolling.

So I threw myself into computer science study. With PyLadies, and my last quarter of school to complete my French degree and Math minor, I busted the proverbial it to get a job, which I did, literally the day I was done with my final FINAL exams, hopefully ever : )

Interne-towne

The job started as an internship with lots of hands-on pair coding with my boss, “how would you solve this problem,” “let’s refactor together,” along with some “hey would you reach out to this person to sponsor (x),” which was great! I got to use a way nicer computer (macbook O2) than I had (a lovely old giant brick of a PC laptop on which I installed Ubuntu 12.04), and I finally started using git at the command line. It became pretty quickly evident that we worked together fabulously, and we started to think about what kinds of projects we could do together, so we started working on new ideas, largely centered on education. It was, actually, an incredible collaboration.

We thought topics and people for Security in Python, Data Science in Python, Twisted, Django From the Actual Beginning, and a few more that I’m sure I’m missing – it was rather a fire-hose of ideas! My boss was the kind of person who had six good ideas before breakfast, and it was a fast-paced, sometimes stressful, REALLY productive space for six months.

While there, I learned git to a granular degree and now lead a monthly workshop on it and plan to lead/teach the PyLadies annual course on it as well, and while I didn’t improve my Python chops much, I learned a lot about how computers are really working, under the hood – well, under a relative hood, I got into no hardware, lol, not at all. The os and sys modules, jeez! I’ve been saying lately that it’s those two modules that turn Python from an expensive calculator into something really powerful.

Codecademy played a little role, too, as you can see if you search the javascript tag on here – while I don’t code in JS, it’s a fairly ubiquitous language & I’m glad to have some familiarity with how it handles different kinds of problems.

I also, god-willing, learned a bit of project & people management the hard way. I don’t ever want to do that again, hooray! It’s good to know, especially considering that women are often directed from engineering career paths to soft-skill positions – now I know what to push back against.

Tutorial Creation

We settled into a Python tutorial with a local $TOPIC_IN_PYTHON expert, and worked really hard on outlining, scoping, creation, refinement, refactoring, presentation, program executability, git monsters, project jupyter/pip/virtualenv & dependency concerns, and so many other logistical issues.

After we flew to the place to film it, the company we’d signed the contract offered a counter-contract, letting us off the hook, and decided not to publish it after all. While honestly heartbreaking, there was a serious amount that we all learned.

UNEMPLOYMENT OH NOES

While scrambling to find a new gig after my internship ended, I stumbled on a number of really awesome opportunities, and though I only was offered one – obviously I stopped looking once I got an offer – I met and now keep in touch with many of the folks I interviewed with, because while it didn’t work out, these are all really neat people at really cool companies, so that has been really validating.

I was strictly unemployed for all of two weeks before I got an offer from Puppet, where I’ve been since mid October, and totally in love with my job. 2014! WOW.

2015?

This year, I will be learning the ins and outs of system administration, I’ll become Puppet certified, I’ll learn Ruby, and I’ll be dipping my toe into web app development as well, all while being part of the neatest leadership team of ladies ever with PyLadies PDX. I think I’d like to get Red Hat certified, or close to it, as I’ve come into Puppet without any sys admin background, and that’s something I’d really, really like to be good at at the new spot. I continue to be amazed by how well I am treated in my new career and how much I can do for people, even while clawing at so much more – frustration at the not-knowing-enough is a constant underlying anxiety for me at the new job, and I suspect that will continue for some time, and which in fact is a good thing. Studying math, and trying to stay aware of what’s Going On in tech, is a pretty good primer for the huge amounts of Not Knowing involved in working in tech, for probably at least the first few years.

WHOO! Onward and upward, or as they say, “Up and to the right!”

Employment and Education

Hello friends! TIME HAS PASSED. But here you are again! This day finds me employed, after a long struggle. Really, I’ve been job-seeking since January, though well in advance of when I needed to, as my graduation date was March of this year. Believe me, I am still fuzzily post-graduation, extremely happy to have no far less homework than while earning my degree (en français, bien sûr !).

Understanding that confirmation bias makes fools of us all, about two months ago I changed my resume (do I write résumé? seems soooo new yorker snobbish, though it is correct) in what may have been a crucial way. My tech recruiter friend gave me some terrific and honest (read: intense) feedback on my R/CL and told me to cut out the “References available upon request” line, because duh, everybody knows that and it just takes up space. For a few weeks, I had it removed entirely. Then, I did something rather bold, and added the following snake-oil-style pitch toward the bottom of my cover letter:

“But don’t take my word for it! Just ask person_1, the leader of the free world, or person_2, the founder of Mars, or even person_3, the inventor of Post-Its! Every one of these folks is happy to -brag about- be a reference for me, so please, contact them!”

And I got a call, from an awesome company that I have always been too afraid of applying to, thinking that the folks that work there are a special kind of brilliant & that I wouldn’t have a chance in hell at actually working there. One of the reasons, other than my qualifications, that they said they called, was because of one of those people who I’d listed in that section.

Typically, references are a very late game process in the hiring world. Why bother calling references, a time-consuming and very personal (and personalizing!) process, if your candidate hasn’t even made it through a phone screen and an interview or two? In other words – why call references unless everybody is serious? But the fact that I put a few folks on there who wanted to vouch for me made a huge difference. And Portland is really so small and the scene is so focused that the names are fairly well-known. That wasn’t an accident, but I met these great people naturally, by getting out, participating quite heavily (and earnestly!) in PyLadies, and making friends with the people around me.

After two phone interviews, a task, an all-day interview, and a few (totally transparent!) hiccups, I was offered the job at Puppet Labs as a support engineer, and I feel so lucky, I have to keep from gushing about it. I left my stable, lifer career nearly four years ago to do ambiguously Better, and yes, Virginia, this is Better.

SO! Now I am LEARNING, learning learning learning! I’m still having a hard time reading the tickets that come in, but what I am able to do is parse Puppet code, and explain what it is and does, and how it’s an enormous, Neil Armstrong-style leap over previous (and still very widely used!) server management technologies. I’m pretty sure I’m in the right industry, guys, as this is really cool to me. Puppet is a company I am extremely excited to work for, for many reasons, not the least of which is getting to know the complex, technical, and awesome product. I keep a notebook on what I’m learning, and I fill several pages a day. Future blog posts will probably just focus on Puppet stuff, unless I get a chance to work on some recreational stuff. Woo-hoo!

And one last thing: if you know me, you know my absolute most highly recommended piece of advice to those looking for jobs: start a blog. Start a blog, start a blog, start a blog. Don’t wait til you code every bit by hand, don’t wait til everything is Perfect, just go to wordpress or blogspot or whatever, and start a blog. Nearly everyone I’ve interviewed with has mentioned it. Fear not about seeming stupid, because you’re brilliant.

Ok – going to cut this off before I get weepy/proselytizey/we-are-the-world-y. GOOD THING.

Meal Planning

Ok, while deciding what recipes I would shop for now that I’m unemployed (because obviously when you’re unemployed it is time to COOK ALL THE TIME), I decided to make a lil Python program that will let me enter recipes, and at a point in the future, probably after I learn regex, make it fully searchable. I’m already separating things out for ease of searching – like lists both with and without quantity indications.

Problems I’ve solved: a) making a new file name out of whatever the user enters, and b) setting a var for the long combination of dir, name & .txt that I’ll use in a few different places throughout the program

global new_filename
new_filename = "recipes/" + new_recipe + ".txt"
open(new_filename, 'a') # 'a' for append

Iterating through a list which adds to another list with some clever while looping

def ingredient_input():
    ingredient = 0  # is it ok to set this var to a dummy value?  it gets redefined with each loop.
    while ingredient != 'DONE':
        print "what is the ingredient?  no measurements yet, please.  type DONE"
        print "in all-caps if no more."
        ingredient = raw_input("> ")
        if ingredient == 'DONE':
            ingred_check()
        else:
            ingredients.append(ingredient)

And there are plenty of problems I haven’t solved yet. Will write more soon! Unemployed coding is fun, ha ha! “Hmm, let me chew on this juuuust a bit more… oh my god how did 3pm get here” etc etc : )