Originally featured on /r/gamedev
I recently started working in games again, joining my good friend and former coworker from PopCap/EA to work on his dream voxel-based limb-slicing 3rd person swordfighting game (our latest screenshot Saturday).
Coming home to games from a stint back in the web world, I'd seen how continuous integration and delivery (CI/CD) has become a nearly ubiquitous part of development teams' workflows over the last 5 or so years. I've been extremely heartened to see how far CI has come for game development as well. Unity has put together a fantastic offering with their Cloud Build, and there are some tutorials cropping up in the last year about how to set up quick, distributed builds for games using Jenkins, GitLab CI, etc.
We recently took the plunge and spent some time to set up CI/CD for Clone Drone, so I wanted to share some lessons learned while they're fresh with you all and do a series of posts about game build infrastructure and continuous integration and delivery. For the benefit of folks wondering "wtf is CI/CD", I'm going to start with some high level thoughts on what continuous integration / deployment is, who it's for, and how to think about the time investment in getting it set up.
What is CI/CD, and why would you spend the time to set them up?
Continuous integration
Continuous integration ("CI") means every commit (or some subset of them) to git/SVN/Perforce/etc auto-triggers builds of your game. This lets you do stuff like:
- Run unit tests (NumberToVerbalString(3) == "three") to ensure core logic still works as expected.
- Run integration (aka "automated") tests that ensure large components of the game still work. E.g., click “play”, beat a level, level should be saved as completed. This automates that “sanity check” you ought to do periodically.
- Monitor build sizes and durations over time. Why's the game build 30mb larger now? Why's the game taking 20 minutes to build? Let's look at the graph!
- Notify an engineer quickly when they break the build. As anyone who has worked on a team of more than 5 engineers knows, "DON'T BREAK THE BUILD" :) -- it's much less stressful to have a computer tell you that.
- Package up signed versions of various configurations of your game — properly signed build zips for each platform, a Steam-specific build, a build with QA tools, maybe even a demo (something we're considering now that there's a way to pump one out in CI).
Running these builds often makes it much easier to chase down the origin of even subtle breakages. It's much faster to do a binary search through old uploaded builds to pinpoint a nasty bug than having to git checkout && run builds for each thing to test.
What do you do with an auto-built build of the game?
This gets in to what's often called continuous delivery (the "CD" in CI/CD). Teams will often have their system do stuff like:
- Upload builds to S3, Dropbox, etc., so you have a history of your builds to pull down whenever you'd like.
- Send a version of your build to a Steam beta branch for your QA friends to test — or in the case of games like Subnautica—actually let Early Access players try out auto-generated nightly builds.
- Upload your game to itch.io using their fantastic butler tool.
- If you're working an iOS/Android game, send builds to TestFlight or HockeyApp to let beta testers try out your game.
Is it worth the time / cost of setting it up at all?
The classic lame answer applies—it depends on your situation. But here are some ways to think through that—
- How many builds do you have to publish, how often, and how long do they take you? If you're going to spend an hour per week not developing because you're exporting a build from Unity, it may be worth spending a few days upfront to get this all automated and happening on another machine.
- How many team members are making changes?
- If it's just you developing a game by yourself, you likely have a good idea of when a given bug might have started, and you won't have to worry as much about forgetting to check in some new file on git breaking the build.
- If you're on a larger team, CI/CD greatly reduce the overhead of.
- What would immediate distribution of new changes mean to your QA folks, beta testers, etc.? Could you tighten the loop between fixing a bug and verifying it's fixed? Would immediate design feedback each day help you iterate faster on new levels?
Welp, That's my high level overview. If there's interest, I might write up a bit about how we put together our continuous integration setup (which outputs DRM-free builds for itch.io and auto-uploads builds to Steam for testing and (very soon!) publishing), those of some other large games and applications, and the software/tools/services that you might find will give you the best bang for the time/money spent starting your integration up.
This was fun to reflect on! 'til next time.
May your tests be ever-green and builds un-broken...