What we talk about when we talk about push

Update: This is a historical blog post, push messages have been encrypted since January 2019.

This is the twenty-first post in the 2016 FastMail Advent Calendar. Stay tuned for another post tomorrow.

---

Something people ask us fairly often when considering signing up with us is “do you support push?”. The simple answer is “yes”, but there’s some confusion around what people mean when they ask that question, which makes the answer a bit complicated.

When talking about email, most people usually understand “push” to mean that they get near-realtime notifications on a mobile device when new mail arrives. While this seems like a fairly simple concept, making it work depends on some careful coordination between the mail service (eg FastMail), the mail client/app (iOS Mail, the FastMail apps or desktop clients like Thunderbird) and, depending on the mechanism used, the device operating system (eg iOS or Android) and services provided by the OS provider (Apple, Google). Without all these pieces coordinating, realtime notification of new messages doesn’t work.

All this means there isn’t an easy answer to the question “do you support push?” that works for all cases. We usually say “yes” because for the majority of our customers that is the correct answer.

There are various mechanisms that a mail client can use to inform the user that new mail has arrived, each with pros and cons.

IMAP

Pure IMAP clients (desktop and mobile) have traditionally had a few mechanisms available to them to do instant notifications.

Polling

By far the simplest way for a client to see if there’s new mail is to just ask the server repeatedly. If it checks for new mail every minute it can come pretty close to the appearance of real-time notification.

The main downsides to this approach are network data usage and (except for “always-on” devices like desktop computers) battery life.

Network usage can be a problem if it takes a lot of work to ask the server for changes. In the worst case, you have to ask for the entire state of the mailbox on the server and compare it to a record on the device of what was there the last time it checked. Modern IMAP has mechanisms (such as CONDSTORE and QRESYNC) that allow a client to get a token from the server that encodes the current server mailbox state at that time. Next time the client checks, it can present that token to say “give me all the changes that happened since I was last here”. If both the client and server support this, it makes the network usage almost nothing in the common case where there’s no change.

Battery life can become a problem in that the system has to wake up regularly and hit the network to see if anything happened. This is wasteful because on most of these checks you won’t have received any mail, so the device ends up waking up and going back to sleep for no real reason.

IDLE

To avoid the need to poll constantly, IMAP has a mechanism called IDLE. A client can open a folder on the server, and then “idle” on it. This holds the connection to the server open but lets the client device go to sleep. When something happens on the server, it sends sends a message to the client on that connection, which wakes the device so that it can then ask what changed.

For arbitrary IMAP clients that do no have specific support for device push channels or other mechanisms, this is usually what’s happening. IDLE works, but has a couple of issues that make it less than ideal.

The main one is that IDLE only allows the client to find out about changes to a single folder. If the client wants to be notified about changes on multiple folders, it must make multiple IMAP connections, one for each folder. This makes clients more complex and may run into problems if there are many connections as some servers limit the number of simultaneous connections for a user.

The other issue, particularly on mobile devices, is that IDLE operates over TCP. This can cause problems when devices change networks (which may include moving between mobile cells), which may break the connection. Because of the way TCP operates, its not always possible for a client to detect that the connection is no longer working, which means the client has to resort to regular “pings” (typically requiring a regular wakeup) or relying on the device to tell it when the network has changed.

IDLE is good for many cases, and implemented by almost every IMAP client out there, but it’s definitely the most basic option.

NOTIFY

To deal with the one-folder-per-connection problem, IMAP introduced another mechanism called NOTIFY. This allows a client to request a complex set of changes its interested in (including a list of folders and a list of change types, like “new message” or "message deleted) and be informed of them all in one go.

This is a step in the right direction, but still has the same problem in that it operates over TCP. It’s also a rather complicated protocol and hard to implement correctly, which I expect is why almost no clients or servers support it. Cyrus (the server that powers FastMail) does not implement it and probably never will.

Device push services

Most (perhaps all) the mobile device and OS vendors provide a push service for their OS. This isn’t limited to iOS and Android - Windows, Blackberry and even Ubuntu and the now-defunct Firefox OS all have push services.

Conceptually these all work the same way. An app asks the device OS for some sort of push token, which is a combination device & app identifier. The app sends this token to some internet service that wants to push things to it (eg FastMail). When the service has something to say, it sends a message to the push service along with token. The push service holds that message in a queue until the device contacts it and requests the messages, then it passes them along. The device OS then uses the app identifier in the token to wake up the appropriate app and pass the message to it. The app can then take the appropriate action.

Deep down, the device OS will usually implement this by asking the push service to give it any new messages. There’s usually some sort of polling involved but it can also be triggered by signalling from the network layer, such as a network change. It’s not substantially different to an app polling regularly, but the OS can be much more efficient because it has a complete picture of the apps that are running and their requirements as well as access to network and other hardware that might assist with this task.

FastMail’s Android app

Notifications in our Android app work exactly along these lines. At startup, the app registers a push token with FastMail. When something changes in the user’s mailbox, we send a message with the push token to Google’s Cloud Messaging push service (or, for non-Google devices, Pushy or Amazon’s Device Messaging services) to signal the change. This eventually (typically within a couple of seconds) causes the app to be woken up by the OS, and it calls back to the server to get the latest changes and display them in the notification shade.

The one downside of this mechanism is that its possible for the message from the push service to be missed. Google’s push service is explicitly designed to not guarantee deliery, and will quite aggressively drop messages that can’t be delivered to the device in a timely fashion (usually a couple of minutes). This can happen when the device is off-network at the time or even just turned off. For the reason, the app also asks the OS to wake it on network-change and power events, which also cause it to ask our servers for any mailbox changes. In this way, it appears that nothing gets missed.

FastMail’s iOS app

The FastMail iOS app works a little differently. One interesting feature of the iOS push system is that it’s possible to include a message, icon, sound, “badge” (the count of unread messages on the app icon) and action in the push message, which the OS will then display in the notification shade. In this way the app never gets woken at all. The OS itself displays the notification and invokes the action when the notification is tapped. In the case of our app, the action is to start the app proper and then open the message in question (we encode a message ID into the action we send).

This is somewhat inflexible, as we can only send the kinds of data that Apple define in their push format, and there’s arguably a privacy concern in that we’re sending fragments of mail contents through a third-party service (though you already have to trust Apple if you’re using their hardware so it’s perhaps not a concern). The main advantage is that you get to keep your battery because the app never gets woken and never hits the network to ask for changes. It’s hard to get more efficient than doing nothing at all!

Since iOS 8 its been possible to have a push message wake an app for processing, just like Android and every other platform. A future release of our iOS app will take advantage of this to bring it into line.

iOS Mail

The Mail app that ships on iOS devices is probably one of the better IMAP clients out there. Apple however chose not to implement IDLE, probably because of the battery life problems. Instead they do regular polling, but the minimum poll interval is 15 minutes. This works well and keeps battery usage in check, but is not quite the timely response that most people are after. When used in conjunction with their iCloud service however, iOS Mail can do instant notifications, and its this that most people think of as push.

It works pretty much exactly like FastMail’s Android app. Upon seeing that the IMAP server offers support for Apple’s push mechanism, the app sends the server a list of folders that its interested in knowing about changes for, and a push token. Just as described above, when something changes the IMAP server sends a message through Apple’s push service, which causes the Mail app to wake and make IMAP requests to get the changes.

The nice thing about this for an IMAP client is that it doesn’t need to hold the TCP connection open. Even if it drops, as it might if there’s been no new mail for hours, it can just reconnect and ask for the changes.

Of course, this mechanism is limited to the iOS Mail app with servers that support this extension. Last year, Apple were kind enough to give us everything we need to implement this feature for FastMail, and it’s fast become one of our most popular features.

Exchange ActiveSync

One of the first systems to support “push mail” as its commonly understood was Microsoft’s Exchange ActiveSync, so it rates a mention. Originally used on Windows Mobile as early as 2004 to synchronise with Exchange servers, it’s still seen often enough, particularly on Android devices (which support it out-of-the-box). There’s a lot that we could say about ActiveSync, but as a push technology there’s nothing particularly unusual about it.

The main difference between it and everything else is that it doesn’t have a vendor-provided push service. Ultimately, the ActiveSync “service” on the device has to regularly poll any configured Exchange servers to find out about new mail and signal this to any interested applications. While not as efficient as having the OS do it directly, it can come pretty close particularly on Windows and Android which allows long-lived background services.

Calendars and contacts

In October we added support for push for calendars and contacts on iOS and macOS as well. In terms of push, they work on exactly the same concept as IMAP - the app requests notifications for a list of calendars and addressbooks and presents a push token. The server uses that token and informs the push service, which passes the message through. The OS wakes the app and it goes back to the server and asks for updates. There are some structural differences in the this is implemented for CalDAV/CardDAV vs IMAP, but mostly it uses the same code and data model as the rest.

The future

Sadly, the state of “push” for mail is rather fragmented at the moment. Anything can implement IMAP IDLE and get something approximating push, but it’s difficult (but not impossible) to make a really nice experience. To do push in a really good (and battery-friendly) way, you’re tied to vendor-specific push channels.

We’re currently experimenting with a few things that may or may not help to change this:

• JMAP defines a protocol for push, using either Server-Sent Events or a protocol not unlike the mobile vendor push channels. The way JMAP push works is that on login, the client sends the server a URL for a push service, rather than a vendor-specific “push token”. The server will send simple “something has changed” messages (matching the JMAP data model) to that URL when something happens. The details of how that message gets from the push service to the client is outside of JMAP’s scope, but what it means in theory is the user could choose a push service to use (perhaps even their own).
• We’ve thought about (but have no currently plans to implement) an IMAP extension that would do largely the same thing - allow a client to send a URL to send change notifications to.
• We’re playing with additional Sieve notification mechanisms that will send new mail notifications to various services, including Pushover, Slack and IFTTT. These are not quite the same as application push, as they are based on user mail rules (filters), but they’re another attempt to find out what it looks like to receive information about your incoming mail flow outside of your mail application itself.

Time will tell if these experiments will go anywhere. These are the kind of things that require lots of different clients and servers to play with and see what works and what doesn’t. That’s not something we can do by ourselves, but if you’re a mail client author and you’d like to be able to do better push than what IMAP IDLE can give you, you should talk to us!