Dec 10: Sunsetting Pobox

This is the tenth post in the Fastmail Advent 2024 series. The previous post was Dec 9: Building a blog. The next post is Dec 11: Meet the team—Marc.

On November 12th of this year, our service Pobox was finally merged into Fastmail. This had been inevitable since Fastmail’s acquisition of Pobox in 2015, but we’d put it off over and over. In the end, it took a lot more time than we expected, but it paid off. The cutover went well… but it was still just a little bittersweet, ending a long era. Pobox was around for thirty years. In Internet years, that’s an eternity. Seeing something so long-lasting cease to exist as itself is a bit, well, confronting.

It hit home for me, personally: I’d been working on Pobox for nearly 20 years, and the shutdown of Pobox meant turning off hundreds of thousands of lines of code I wrote or maintained. A professional programmer does well to avoid identifying too much with their work product, but on some level it’s hard to avoid. Still, it was better to do it myself. I had a good knowledge of both systems and felt like I was best placed to make things as seamless as possible.

So, how’d it all go? I’ll walk you through it.

What even was Pobox?

Pobox was, at its heart, an email forwarding service. It was started by Meng Wong and Helen Horstmann-Allen in 1995, while they were both still in college. College students in the 90s had email addresses, but they knew they wouldn’t last. Eventually you’d leave school and you’d need to tell everybody your new address. The same thing went for people who had email at work. Pobox was the answer to this problem: you’d get an address with Pobox and set it up to forward your mail to whatever your current “real” address was. Unlike Fastmail, Pobox didn’t include mail storage by default. You could get it, but it was an add-on.

Broadly speaking, Pobox and Fastmail were very similar. Users would have one or more email addresses, and those addresses would deliver to an inbox or to another address. Spam got filtered out, and users could write their own mail filters. One “group” could have a bunch of users on it. Users could send mail through relay servers.

So what was the problem?

Yes, broadly speaking, Pobox and Fastmail were very similar. In the fine details, though, everything was just kind of different in tedious little ways. Email aliases were per-group in Fastmail, per-user in Pobox. Spam filtering exceptions worked differently. The rules on sharing domains were different. In short, no feature really worked the same in both services, and every exception had an exception.

Obviously, maintaining many of the same features (but differently) in two places took a lot more time and effort than just maintaining one version. The lion’s share of development always went to Fastmail while the number of staff with expertise in Pobox dwindled. We kept training new support staff on Pobox, but new programmers almost never touched it. Pobox had emergency support staff, but no full time developers, and it showed, with nearly no changes of any kind after 2015. That didn’t just mean they weren’t getting cool new features. It meant they weren’t getting as many deliverability improvements or updates to service monitoring.

In order to keep Pobox users happy, online, and connected, we were going to have to turn them into Fastmail users. We wanted to make this as seamless as possible, which was going to be real work. Remember: everything was different enough to make “just import them to Fastmail” not work. Still, the goal was that cutover would happen without significant downtime, and without users needing to reconfigure their mail clients, change passwords, or (worst of all) fiddle with their DNS.

This was going to be a complicated piece of work, and risky, and was going to require significant knowledge of both Pobox and Fastmail internals. Also, it just wasn’t going to be exciting work day to day. The final payoff would be great, but otherwise it would involve writing a lot of fiddly and risky code that was just going to get deleted later. So we put it off a long time, even though we’d known since 2015 that we’d have to do it eventually. In 2023, we finally started in earnest.

The Upgrade Process

Back when Fastmail acquired Pobox, the first thing we changed was the Pobox mail storage. Both Fastmail and Pobox used Cyrus IMAP, but Fastmail ran it at larger scale and with better tooling. Very early on, we created a secret Fastmail user for each Pobox Mailstore account and moved mail out of Pobox and into Fastmail. This also let us replace Pobox’s webmail with Fastmail’s best-in-class UI.

It didn’t replace almost anything else, though: IMAP and SMTP connections went through Pobox. Pobox handled the mail forwarding. Settings and billing were all in Pobox. That meant that those “secret Fastmail users” had all kinds of settings that nothing used. The Pobox Upgrade Project (known internally as PUP) used these as the starting point: they would be configured to act almost exactly like their Pobox owners, and then they’d take over.

First, we created these Fastmail users for all the non-Mailstore Pobox accounts. This was pretty painless. Next, we merged these users into customers to match the billing setup in Pobox. That’s where the pain began. Fastmail supported merging multiple users into one customer (to get just one bill)… but not when both of those users are free and have no expiration date. We had to get into the billing system and add ways to override the safety mechanisms keeping us from breaking the rules that we so desperately needed to break. I won’t provide a litany of every such complication, but: there were plenty. Each ported feature involved one or two big mismatches, one or two edge cases, and one or two places where everything would work just fine, if you just bypassed all the business logic briefly.

For months, the process moved forward feature by feature. At Pobox, the “migration planner” kept learning how to describe more and more of each Pobox account’s configuration in terms of Fastmail. The planner wrote out new plans constantly, and those were shipped from Pobox to Fastmail. On the Fastmail side, the “migration executor” would take those plans and reconfigure things on the matching Fastmail users. Every week, the Fastmail users were configured more and more like their counterpart Pobox users. The big question was: when could we cut over?

Cutover

When making a big change, we generally like a staged rollout. First, we apply it to some of our internal test customers. Then we apply it to ourselves. Then one percent of users, then five… you get the idea. We’re looking to reduce the damage that a bug can cause. The Pobox cutover was exactly the kind of big (huge!) change where we’d like roll things out in stages.

Unfortunately, it wasn’t going to be that simple. Email is routed by domain. You can’t really send one user’s mail through one system and another user’s mail through another mail system. Or, you can, but it requires building a third system that sits in front of both, which is just another thing that can go wrong!

We decided that all users would get cut over at once. To mitigate that big change, we further decided that we’d cut over different components over time, as much as possibly invisibly to the user, like this:

• First, anybody connecting to IMAP or POP was connecting directly to Fastmail, not Pobox.
• Later, DNS for our domains (and user domains) was moved to Fastmail.
• Later still, anybody sending mail through Pobox was actually connecting directly to Fastmail, not Pobox.

Each of these cutovers was a big change that we’d been able to test well in advance, and each one turned up one or two weird edge cases that took a little time to fix. Doing them one at a time let us focus and get things right. As far as I know, nobody reported noticing that any service had moved to Fastmail.

Finally, the big cutover day was set for November 12th. That’s when we’d have all mail destined for Pobox customers start hitting Fastmail servers, and when we’d make pobox.com start taking people to Fastmail. We had a big checklist, and the plan was that we’d start our day at 8:00, run a series of programs, watch some logs, and be done around lunchtime. That was the prediction if everything went perfectly. In reality, we weren’t done until about three in the afternoon, and we did spend a decent amount of time responding to unexpected cases. Mostly, though, it went smoothly and we were done by the end of the day.

The goal had been to avoid a painful reconfiguration or transition process for users, and I think we did pretty well on that front! I’m proud of that. Pobox is how I got involved in working on email, and it’s been one of the major things I’ve worked on for twenty years. Working on Pobox made me a better programmer and helped me understand what it means to be a good internet citizen and a good steward of customers’ data. Astoundingly, most Pobox customers have been with Pobox at least as long as I have. I’m pleased we could bring them smoothly into Fastmail, where we can keep trying to deliver the best email service around.

Everybody at Fastmail contributed to the success of the project, but I think I should especially shout out two teams. Team PUP (meaning Mark Jason Dominus and Matthew Horsfall) worked on the project full time, dealing with weird legacy systems, obnoxious edge cases, and plenty of complexity arising from wiring together two incompatible systems. It was a bit of a slog, and they did great. Also, our support team (Team SUP!) were invaluable in this project. They wrote the Guide to Fastmail for Pobox users, they provided feedback on what decisions would help make things easiest on users (and on them!), and they took care of our users whenever things weren’t quite as seamless as we’d hoped. One of the most common concerns I heard from Pobox customers was “the support won’t be as good.” The great news is that it’s just one support team that’s been supporting both products, so the same great support is going to be there for you.

Thanks for a few good decades. I’m looking forward to what comes next!