Dec 22: CardDAV beta release

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 21st December was about our file storage system. The following post on 23rd December promotes our standard for better email.

Technical level: medium

After more than a year of anticipation we’re very happy to announce today that we’re releasing CardDAV support into public beta test.

CardDAV is a protocol for reading, writing and synchronising contact data. It’s built into iOS devices and available on Android with an inexpensive third-party application. If you’ve ever wanted to have your FastMail contacts available on your mobile device (and vice-versa), then this is exactly what you want.

Obviously, since this is a beta, there are still a few pointy edges and non-working bits. The most notable thing is that the beta is currently only available to personal accounts, not to business or family accounts. This is because support for shared contacts is not ready yet and there’s some potential for data loss and inconsistent behaviour if you try to use shared contacts without proper support. We’re working hard on finishing shared contact support and hope to make the beta available to business and family accounts within the next couple of months.

CardDAV is only available to Full account levels and higher. Member, Guest and Lite accounts will need to upgrade to be able to use CardDAV.

So now all the disclaimers are out of the way, you can sign up for the CardDAV beta. Instructions for connecting your client are still under development here: https://beta.fastmail.comhttps://www.fastmail.help/hc/en-us/articles/360058752834-Set-up-Fastmail-on-your-device.

The contacts story

An address book is a fundamental component of any mail system and FastMail has had one almost since the beginning. It’s always been stored in the MySQL database and available through the web client. For much of its history it’s been confined to the web client. A few years back we did add a read-only LDAP interface, which is useful for desktop mail clients that could support LDAP address books. It works fine, but being read-only severely limits its usefulness. Some time later mobile devices happened, and it became clear that something else was needed.

In 2011 the CardDAV protocol was published, largely developed at Apple to allow device contacts to be synchronised with a server. The protocol is very similar to the earlier CalDAV protocol (which we also use for our calendar) which is good as it allows us to share a lot of code between our calendar and contacts system.

Towards the end of 2012 we started to seriously appreciate the need for both an integrated calendar and device contacts syncing. We weren’t the only ones, as the Cyrus project had started to add support for CalDAV and CardDAV to the Cyrus mail server. We looked at a few options for calendar and contacts and decided that we would implement both on top of the support being baked into Cyrus, and work began in earnest. We decided that calendar was more important because we already had a contacts system and while it wasn’t perfect we preferred to focus our engineering effort on a clear gap in our product lineup. That work took the best part of a year, and we finally released the calendar to production in June 2014. At this point we were able to focus on CardDAV-based contacts.

The actual CardDAV part of this work is actually fairly simple. Unlike CalDAV, the backend server (Cyrus) doesn’t really need much special knowledge. It mostly just saves and loads contact cards as required. Calendar entries are more complicated; the server needs to know about timezones, recurring events, alarms, etc. CardDAV is much easier and if all we had to do was ship CardDAV support, we probably could have done so months ago.

The thing that made it more difficult came from the fact that we already had a contacts system and plenty of code fairly tightly integrated with it. It’s more than just the two user interfaces. The mail delivery pipeline also makes use of user contacts for spam whitelists and distribution lists, so we needed to teach these systems about a whole new storage system for contacts. Up until this time they had simply hit the database for this information. To make matters worse, we always knew that we’d need to roll out CardDAV to users gradually which meant that both the UI and the delivery code needed to be able to work with either. In short, we needed to abstract away the implementation details of the contacts storage, which took a few months to build, test and deploy. We ended up with a nice abstraction based on the JMAP getContacts/setContacts model, with a database provider behind it.

The next step was to write a CardDAV provider for our contacts abstraction. That was actually pretty easy because most of the code needed to access DAV resources was already available from our CalDAV work.

The last piece of the puzzle was the actual data conversion layer. The existing contacts system has a data model that doesn’t match up perfectly with the vCard format used by CardDAV, so we had to develop a mapping. Most of the fields have a 1:1 mapping (addresses, email addresses and phone numbers). What we call “online” fields, however, do not. Our “online” field group includes URLs, Twitter handles and chat IDs. vCard doesn’t group those the same way but more annoyingly, it doesn’t have a standard set of fields for representing these. It took a long time to develop and test a mapping that works most of the time. It’s going to need improvement as we go but it’s not bad for now.

What’s next

The next few months will include a lot more testing, polishing and responding to user feedback and obviously completing the business and family support. That will bring us to a full release where everyone will be quietly and transparently migrated to the CardDAV backend. We can then start to clean up a lot of old code, always a nice thing to do.

If you’re trying the CardDAV beta test, we’d love to hear what you think. Let us know on twitter or by emailing carddavbeta@fastmail.com.