Dec 23: JMAP — A better way to email
Post categories
CEO
This blog post is part of the FastMail 2014 Advent Calendar.
The previous post on 22nd December was the long awaited beta of contact syncing. The final post on 24th December is about our work environment.
Technical level: low
You can reproduce the demos by signing up a trial account at FastMail and trying for yourself!
Protocols
FastMail started with IMAP in 1999 when it was still quite new. We have been involved in the standards process, and have watched while non-open protocols led the way on features, usability and reliability. We pride ourselves on the standards compliance of our server, but we have been frustrated by the lack of progress in the third-party client experience available to our users.
The fragmentation of server support for newer IMAP features means that clients either have multiple implementations of everything, with the complexity and bugs that involves, or fall back to lowest-common-denominator behaviour. Extending IMAP further just makes this situation worse.
Having a separate protocol (SMTP) for sending email, running on a different TCP port and with potentially different credentials provides opportunities for partial failures, where a client can send but not receive or vice versa. These issues can be caused by firewalls, by temporary maintenance on the server, or just misconfiguration.
Regardless, for the end user the result is confusion. This is a frequent support problem for anybody with non-technical users, and the addition of CalDAV and CardDAV — everybody wants calendar and contact sync these days — means yet another failure mode in the mix.
FastMail’s protocol
When we built our own client and API we had the benefit of years of experience. I had rewritten the Cyrus IMAPd internals with strong locking to allow reliable QRESYNC support. We benefited from the work done by the LEMONADE working group for mobile email.
We also hooked in to MBOXEVENTS and FUZZY SEARCH, building on standards in the backend to create a great experience for our users. We added non-standard conversations support to our server, and there is discussion on the protocol lists about standardising that in a way which is compatible with gmail and other implementations.
Our servers are in New York, and our developers are in Australia. Even on a good day, the ping times are over 230 milliseconds. Anything more than the absolute minimum number of round trips is felt very keenly.
The end result is a very efficient, stateless, easy to use JSON API which already provides a great experience for our customers.
A proliferation of protocols
We are by no means the only people to have this idea. Many companies are building APIs to access email. Here’s just a few of them:
None of these APIs are designed with a primary goal of enabling 3rd party clients to work efficiently, with few round trips and low bandwidth use, and they are all limited to a single vendor. To inter-operate, everything falls back to speaking IMAP and SMTP.
Enter JMAP
JMAP is FastMail’s protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them - making it easy for developers to work with.
JMAP is friendly to mobile devices. By batching multiple commands in a single HTTP request, and having an efficient update mechanism, the radio works less, and battery life is increased. By using the platform push channels, JMAP avoids having to hold its own connection open.
JMAP is friendly to servers. A stateless protocol, there’s no need for the server to maintain a version of the mailbox view that’s out of sync with the current state, as IMAP does, just so that clients can use integer offsets to refer to messages.
JMAP is friendly to the network. Unlike IMAP which can send an unbounded number of unsolicited updates, in JMAP you explicitly ask for the fields you are interested in, and the update command can contain a limit — if there are too many changes, return an error and let the client load the view from scratch. If you’re not caching the entire mailbox locally, then re-fetching a few pages of index listing is better than getting 100,000 EXPUNGED lines down the wire.
JMAP is friendly to multiple clients. In IMAP, if you rename a folder or move messages, then the client which initiated the action can update their local cache, but every other client has to re-fetch everything - there is no unique message ID which follows the message. JMAP uses globally unique IDs for messages, so immutable data can be cached forever.
JMAP does everything. Instead of separate protocols with different syntax for sending email than for receiving it (and separate protocols again for contacts and calendars) JMAP combines them all under one protocol. Best of all, the push notification includes all tokens for each service, so you can synchronise changes to your entire mail account plus your contacts and your calendar in a single round trip — up to date immediately, the way it should be.
Open everything
The JMAP protocol is totally open and unencumbered.
FastMail commits to provide a reference server and reference client, as well as maintaining the specification document in collaboration with others and keeping an up-to-date test suite to allow verification of implementations.
Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP. If the proxy is run somewhere close (in network terms) to the legacy servers, then JMAP will be a better experience than speaking to the servers natively, particularly if you’re on a slow network or the other side of the world.
As more servers start talking JMAP natively, the baseline for interoperability will be raised beyond IMAP+SMTP.
Come and join us at jmap.io and on the mailing list. Together we can build a better way to email.