Goodbye old.fastmail.fm

Summary

In early 2009 we rolled out an updated web interface to all users. This is the interface you currently see when you login at http://www.fastmail.fm as most users do.

To give users time to transition, we continued to let people login to the old pre-2009 interface if they wanted to by going to the web address http://old.fastmail.fm. We’ve continued to support this for the last 3
years, but as only a few users are still using this interface, we decided to shut it down. For the last 3 months there’s been a prominent message each time you logged into http://old.fastmail.fm that noted this, and we’ve now fully shut down http://old.fastmail.fm.

Important point: This only affects users that were explicitly going to http://old.fastmail.fm to login. Users that use the regular interface at http://www.fastmail.fm (the vast majority) are completely unaffected.

The description below is a detailed history of the old interface and includes technical details about how much things have changed since 2009 and why maintaining http://old.fastmail.fm is no longer feasible.

Goodbye old.fastmail.fm

It’s been a long road, but the old FastMail web interface has finally reached the end of its life.

You can always access your email at https://www.fastmail.fm/ or try our beta site at https://beta.fastmail.fm/.

If you want to stop reading here, the things you need to know are
security concerns and it was about to break anyway. Two good reasons why now is the right time to shut the old interface down.

If you want to know some of the technical background and the technologies that we have moved through over the years, read on!

A new infrastructure

Looking back through our version control history, my very first commit was on 2004-09-20! The original web interface commits are from early 2000, though it was started before then.

We switched version control systems at some point during 2005 from CVS to Subversion, which made branching much easier - but imported all our history, so we can still look back at those early changes.

One of our major branches was a huge infrastructure switch from Redhat
7.3 to Debian 3.1 (sarge), which we worked on throughout the second half of 2005. This was all merged back into the main branch, and we converted everything over in early 2006.

</2006/01/02/one-web-server-now-running-new-infrastructure-code/>

We upgraded to Debian 4.0 (etch) during May 2007, soon after it came out.

A new interface

In 2008, Neil Jenkins (who is so awesome Opera hired him even before they had decided if they were going to buy FastMail) worked as a contractor over the summer to design a more modern web interface which would take advantage of the new features in web browsers.

We branched the code, and it diverged quite considerably. Features like cross folder searching required major internal datastructure changes, and the new interface had hooks all through the code. Our plan was always to retire the old code eventually.

We released the new interface to beta at the end of 2008, and rolled it out to everyone in 2009.

</2008/11/27/help-beta-test-new-web-interface/>

</2009/02/17/new-interface-being-rolled-out/>

An incompatible upgrade

Then in 2009, Debian 5.0 (lenny) came out. Lenny shipped with apache2 and mod_perl2, and no longer supported apache 1.3 or mod_perl version

1. We put quite a lot of work into porting our codebase forwards to apache2. Since “old” was going away soon, we didn’t duplicate the work there.

So we installed the new web servers on lenny, and kept a couple of servers called “oldweb” still running etch. It’s amusing now to remember all the hoops I jumped through to allow automatic installation of either system.

About this time we also had machines with enough memory that 32 bit address spaces were wasteful, particularly on the IMAP servers. We moved to running 64 bit kernels with 32 bit userland.

New hardware

In 2010, the Opera sale happened. One of the early steps was to replace some of our aging hardware with equipment that was better understood and supported by the Opera sysadmin department. This meant a new bladecentre for the non-storage systems (including web)

For a little while I had two blades (redundancy!) running “oldweb” code. That’s a huge amount of very under-utilised resource.

And, to be honest, managing new blades with ancient OS was a pain. Things didn’t work well. The configuration tools we built for the new hardware didn’t run on etch.

When we moved to Debian 6.0 (squeeze) and at the same time went fully 64 bit, it was time to do something about “old”.

We also moved version control systems AGAIN in late 2010 - from subversion to git. The old web servers were left on subversion, because they weren’t getting much in the way of changes any more. One more “split” in how things were done.

Fully virtual

Rather than having to support “real hardware”, I built an etch virtual machine. Everything else was running squeeze 64 bit, but we still had a full 32 bit etch install path just to support oldweb.

While all this was happening, there were occasional changes required to support changing database schemas, configuration mechanisms, and interaction with other parts of our system. At some point I just took a snapshot of the current tree and started a new git repository so we could archive the subversion server entirely.

Maintaining the virtual machines was a real pain though. They were run in the background on some of the web servers to free up the hardware for more demanding tasks. This meant changing the network interfaces to be bonding drivers, custom configuration, lots of pain. There were occasionally long outages as we changed things and then had to patch oldweb to catch up.

Worst of all, we were maintaining the ENTIRE stack - support daemons, log rotation, pop fetching…

Old lost features over time - we just couldn’t keep them working, so we ripped the code out. Particularly some of the more advanced configuration screens - and everything related to billing.

Single component

In the end the virtual machines were too much work. Our authentication system in particular had many changes under the hood, and it just wasn’t going to keep working. We had a couple of really bad problems with file storage, where we were sure that something “couldn’t happen”, but then it turned out old was still doing things differently. Talking to the wrong databases, running the wrong queries. We seriously considered dropping old at that point, but I wanted to give it a bit longer.

So I build a chroot installation of etch on our web servers, and bind mounted the daemon sockets into the chroot. This allowed us to run just the web interface code itself on the old branch, while running everything else in the modern, managed, outside world. I built a custom init script which could set up all the necessary mountpoints (/proc, /dev, /var/run, even the tmpfs with mmaped caches was shared) - and forward ported more of the code.

This was built with debootstrap originally, but in the end it was getting unreliable even fetching etch packages, so I build a .tar.gz file with the filesystem for the chroot, and a fresh install just unpacked that. As we changed internal config systems, I kept “oldweb” up to date. A couple of commits every month.

So that’s brings us to today. An init script (apache-oldweb), a chroot environment with a snapshot of a Debian etch machine with apache 1.3 and mod_perl version 1 - running perl 5.8. Everything else is perl 5.10 or newer, so I even have to backport some idioms as I bring back the bits which it just can’t live without.

I have done basically all the “keeping old alive” for the past couple of years - for a smaller and smaller set of users who still log in there. Backporting everyone else’s changes as they impacted old.

And etch doesn’t have security support. Hasn’t for ages. Sure it’s in a chroot, but it still has access to everything.

The final straw

But there’s one thing which oldweb can’t survive. We are redesigning how our session management works. There are some great benefits - bookmarkable URLs, remote logout of stale sessions, reduction of password typing on annoying little smartphone keyboards.

Everything will change, and old would have just stopped working. It’s not worth the changes to make it work. Particularly with the larger gap between the two systems as time goes on.

Also, and even worse, old interface is exposed to the wider internet - and it has full read/write access to the database and all emails. If there are security problems, all our users are at risk - not only those who use it directly.

It’s no longer safe, and it was going to break beyond easy repair in a few days anyway. It is time.

Goodbye oldweb