November 30, 2020#college | #technical

Overhauling Infrastructure for CS 61A

I've been working as a tutor this semester for CS 61A. Until about a week ago, my job has been primarily teaching, with a focus area in extra mentoring as well helping with grade management, since I took care of the latter as a TA over summer. However, after a bug popped up in the grading system last week, I volunteered to fix it instead of deferring to our Head of Software like we usually do. Since then, I've turned 61A infra over and improved multiple aspects of it.

The bug was simple: our grade display tool, howamidoing.cs61a.org, had an erroneous line of code that broke the display if we enabled an option to allow students to plan their grades. This was a result of an additional feature someone had added last semester, which actually gave staff access to histogram distributions of student grades. This was valuable info, but in adding this feature, one of the method signatures for displaying grades changed and this change was not propagated to all calling functions.

But that's not the important part. The important part is that fixing that bug got me into a conversation with the Head of Software about how we'd been using Dokku to run some services, but that Dokku was eating up a lot of our Google Cloud credits. Remember this post? Well, I pointed him to an anonymized version of how my build system worked, and he said it was almost exactly what 61A infrastructure needed to get rid of the Dokku machines once and for all.

So I did it. He gave me access to a spare machine on our GCP project that we weren't doing anything with, and I got to work setting up a similar system to the one I was using on my website. Except this time, it was even simpler: our existing buildserver uses Cloud Build to build Docker images, so all I had to do was pull and deploy them, as well as proxy appropriate domains to them. It took about two days to get up and running, and by the end of that week our stateful apps were deploying 50% faster and we were saving hundreds of dollars a month from having killed the Dokku machines!

I didn't stop there, though. One of the hassles in the grading process was collecting student grades from multiple sources like Okpy and Gradescope and updating them on Howamidoing every week. The process took about an hour, and was comprised of a lot of manual labor. In 2020, it felt like far too much unnecessary work to have to manually export, combine, and upload grades for a class that prides itself on in-house infrastructure. So I decided I'd just write my own scripts that could leverage our infrastructure to automatically update grades efficiently twice a day.

The only hold-up here was Okpy's slow export. It turned out that a couple of staff members had already begun work on making those exports faster, so all I had to do was pick up where they left off. Took about two hours, and Okpy exports went from 30 minutes each to <1 minute each. Yeah, you read that right. I combined that with the rest of my grade collection scripts, and the entire process now takes 2 minutes of automatic action. I don't even have to look at the scripts most of the time -- even as new assignments are added and graded, the script automatically picks it up!

One of the other things I started was a wiki for our infrastructure. We currently don't have a whole lot of documentation around our various apps, mostly because the only person actively working on them is our Head of Software. However, he wants to get more people involved, and I want to make our READMEs more visible, so I set up wiki.cs61a.org. This wiki is far from complete, but it compiles all of our existing READMEs into one place. Ideally, next semester I'll encourage all of our staff to get in pairs and tackle an app's worth of documentation. It'd be a nice bonding activity!

In the last week, I've sped up app deploys by over 50% and grade updates by over 90%. This is just the tip of the iceberg. Not only do I love teaching this class, I also love working on its software. I've been rehired as a TA for the Spring semester, and I'm excited to see where that and future semesters on staff for this wonderful course will take me. Cheers!