Dec 21: Fastmail In A Box

This is the twenty-first post in the Fastmail Advent 2024 series. The previous post was Dec 20: How Fastmail uses Fastmail!. The next post is Dec 22: Why we use our own hardware at Fastmail.

They say everybody has a testing environment. Some people are just lucky enough enough to have a separate environment for production. At Fastmail, every staff member can get their own isolated testing and development sandbox. We call this Fastmail-In-A-Box, or more commonly just “fminabox”.

Like many technologists, I learn most by fiddling with things, often breaking them along the way and putting them back together again. With fminabox, we give everyone their own world to break apart and put back together, risk free. This makes it an invaluable place for new hires to cut their teeth, and for existing staff to come up to speed in an area of Fastmail’s stack they haven’t worked on before.

Fminabox is a complete Fastmail deployment on a single host. This includes Cyrus for IMAP storage, Postfix for incoming and outgoing mail, MySQL for non-mail data, our JMAP web API, and all the frontend assets. It also runs the ancillary services we use to monitor Fastmail such as Prometheus, all managed by the same configuration and service management system we use in production. This allows for fast, iterative development with very little waiting time between making a change and seeing the effect, while eliminating most “it worked on my machine” bugs.

We use Hashicorp Packer to create fminabox, following the provisioning scripts we use in production as closely as possible. Who hasn’t made a change to a system where it works going from state N → N+1, but then discovered weeks later that it’s broken when bootstrapping from nothing? Each night we build a new image from scratch. This allows us to catch those types of failures, and to do so while the changes are still front-of-mind in the developers that made them.

Any staff member can tell our chatbot Synergy to box create, and Synergy will handle provisioning a VM in the cloud, set up DNS, and provide VPN configuration upon request. Fastmail continues to eschew the public cloud in favour of our own hardware to run our product, but it turns out the public cloud is really useful for creating test environments.

Fminabox is also a key part of our testing workflow. Fastmail has thousands of tests, from simple sanity compile checks to complex integration tests between systems. We use fminabox with our CI/CD pipeline so every change is automatically tested before it is merged. This was the ultimate progression from developers just running a handful of tests manually, to overnight runs, to fully integrated continuous testing.

As new needs arise, we continue to evolve the infrastructure. A few years ago I was making an improvement to our tooling that balances users between machines in our Cyrus backup system. At the time, fminabox only had a single target that all users were backed up to, so my first step was to add support for multiple backup targets. Only then did I feel comfortable that I could properly test any changes to the tooling.

I’m not the only user, so I asked some other Fastmail staff members “what’s your favourite feature of fminabox?”, and here’s what they had to say.

Fastmail is able to send via externally authenticated submission via OAuth, but Fastmail is also an OAuth provider and provides authenticated SMTP submission via OAuth. We were able to update our test suite to do full end-to-end OAuth authentication with ourself, send an email back to ourselves, and see that this entire path works.

—Rob Mueller, CTO

The best thing about fminabox is that it’s cheap and disposable. If I mess it up, I throw it away and make a new one and act like nothing happened. (The previous solution took hours to create a new box.)

—Ricardo Signes, Head of Special Projects

Although it only takes 5 minutes to setup, inaboxes provide a fully contained sandbox that includes all of our code ready to test. Minutes to build, seconds to tear down.

—Marcus Love, System Engineer

And that’s the story of fminabox. It isn’t perfect but it’s pretty damn good and it helps enable my colleagues to get their work done.