JMAP Open Standard for Email Moves Forward

JMAP, a modern, open protocol that provides a better email experience, is nearly an official standard and in use by many of our customers.

---

In 2014 we wrote about JMAP, the open protocol for modern email which had grown out of the web interface we’ve had since 2011. We believe in open standards, both as a contribution back to the community from what we’ve learned, and as a bulwark against the growth of walled gardens.

JMAP in 2014 was a first draft of a protocol and an idea. JMAP in 2018 is almost finished the standard process, and has solid real-world use.

Let’s have a look at what’s happened over the past 4 years.

Building community engagement

We started the JMAP process in 2014, by talking to everyone we could find — at conferences, online, even visiting companies directly and chatting to our counterparts in their email teams! With these discussions we validated that there was interest in a modern, open standard for email which could give a user experience as good as or better than proprietary alternatives. One year after announcing, we wrote another blog post detailing some of the people building on our first JMAP proposal.

Key features of JMAP were significantly better than the protocols available at the time: push updates for immediate notification of changes, batched commands to reduce latency, and a single authentication and endpoint for related data types to eliminate the support headache of partial authentication failures.

The use of JSON and HTTP as the basis of JMAP was always a key point — it means that people wanting to build something on top of email don’t have to re-implement complex parsers or find a software library in order to get started.

We found plenty of interest, but staff at many of the bigger players said “this sounds great, but we won’t be allowed to build it into our products unless it’s a standard”. Fair enough. We were happy to self-publish it as a standard over at https://jmap.io/, but there was clear demand that it be standardised properly.

JMAP at the IETF

The IETF (Internet Engineering Task Force) is the premier standards body for the internet. We first proposed JMAP to the IETF in late 2016, and a new working group was founded to work on it in early 2017. I wrote last year about our work there so far and the chartering process. In the first year, major changes were made to JMAP. Authentication was removed entirely from the JMAP spec proper, because it’s more general than just one protocol.

In general, the IETF likes separation of concerns. In the same way that an office building might have individual tenancies fitted out frequently, entire floors redone on a less frequent basis, and yet the skeleton of the building outlast all those changes, IETF protocols are built with infrastructure at different levels expecting to have different lifetimes. It is likely that authentication protocols will evolve, and even possibly that transport protocols will change, and the core object structure of JMAP will remain relevant. So it made sense to remove authentication.

Other major areas of JMAP have been completely rewritten. The original structure of setMessages, getCalendarEvents etc gave way to a Noun/verb syntax. The fact that “Message” meant too many things led to it being renamed as “Email”, so we now have Email/set and CalendarEvent/get. Originally, email was sent by adding the mailbox with the special role “Outbox”. We now have an EmailSubmission object which can provide an interface to things like delayed send, undo send, and message delivery status if the underlying system supports them.

The entire Email object was redesigned to allow a middle ground between the really basic client that doesn’t care about email structure at all, and a full service client which needs to understand the entire MIME structure. By adding the middle ground, it’s possible to handle structured email sensibly, use bandwidth efficiently, and barely ever have to care about the raw email.

Finally, there was a separation into a core protocol document and an email specific document. This also allowed us to remove a ton of duplication throughout the specifications, and will make future JMAP extensions easier to write. We expect the next extension to add calendaring, and soon after that a contacts/addressbook module. The delay on addressbook is due to there not yet being a fully specified standard JSON data object for contact records.

The IETF process has definitely made JMAP better — more regular, more general, and more easily implemented in more than just our environment. We appreciate all the effort that members of the working group have put into improving the protocol.

JMAP Core and JMAP Mail are currently at Last Call within the IETF, the working group effort having been finished. We expect them to be published early in 2019.

JMAP at CalConnect

CalConnect is an industry group involved in all things calendaring and contacts. While JMAP itself isn’t being worked on directly at CalConnect, a JSON data model for calendars was already underway. We joined that effort, and then brought the work to the IETF, where it is also close to standardisation in the calendaring extensions working group.

CalConnect is also concerned with the next generation of addressbook formats, and we expect the work on the contacts data model to continue in both venues.

We actually started at CalConnect before going to IETF — we’ve been a member of CalConnect since 2014, and we were aware of them for years before that due to Ken from the Cyrus development team being involved via Carnegie Mellon University.

JMAP at Fastmail

It’s all very well to talk about JMAP in the abstract, but we were also keen to make sure it worked in the real world. We employ developers who work on the Cyrus IMAP open-source email server, which we run ourselves. The Cyrus team, particularly Robert, have fully implemented JMAP as an open source server. Our Perl developers also built Ix, a Perl JMAP server framework done properly, and I slapped together the JMAP Proxy (which is now kinda out-of-date — I hope to finish updating it early in the new year).

Finally, we’ve been working hard on a test suite for JMAP, to help others implement their servers.

But enough about the software, how about the experience! When we created our brand new Topicbox product, we built directly on top of JMAP for the email. We also used JMAP-inspired APIs for the rest of the product experience, so Topicbox’s early users have been on JMAP for over a year now.

For the Fastmail product, we started rolling out a limited beta to our users a few months ago. We had hoped to have it deployed to all users, but there are a couple of final bugs that aren’t ironed out — in particular the non-conversation mode has issues with the fact that an email in JMAP can exist in multiple folders at the same time (which can be used to support a label mode), so the counts are wrong in some edge cases. Still, 30% of our customers have been converted to JMAP already. All new customers are on JMAP automatically, and our new mobile applications which are fully based on JMAP will come out early in the new year.

The future of JMAP

We expect the core and mail specifications to be standards very soon. The JMAP working group at IETF plan to extend our charter to include contacts and calendars, and perhaps notes and files as well. Inside Fastmail, we’ll keep building both our internal and external protocols in the JMAP way, and publishing specifications once we’re happy with them so we can be more of a platform.

If you’re interested in being involved in JMAP, the IETF mailing list is currently the best place to discuss the protocol. Once the specs are in the field, we will find a better place for a user community to build around the protocol. We expect existing users of earlier JMAP drafts will update their implementations once the standard is published, and the large email providers will see the benefit of implementing JMAP gateways to their mail stores.

We will continue to improve the test suite and evangelise JMAP as the best email protocol out there!