Cyrus development and release plans

This is the twenty second post in the 2016 FastMail Advent Calendar. Stay tuned for another post tomorrow.

---

Cyrus IMAPd development

As we mentioned earlier in this series FastMail is a major contributor to the Cyrus IMAPd project. As the current project lead, it falls to me to write about where we’re at, and where we’re going.

Since last year’s post about the Cyrus Foundation Cyrus development has both slowed and sped up, depending what you’re looking at. We haven’t advanced the Object Storage work because nobody was sponsoring it any more. Ken from CMU makes it to our weekly meeting, but his availability to work on the open source code depends on how busy he is with other responsibilities.

So for now at least, Cyrus is mostly a FastMail show, and obviously anything that FastMail needs for our own production system takes priority for our staff, and that’s where our development resources go.

Still, there’s been a ton of work. Looking at commits, well over 10% of the total changes ever happened this year:

brong@bat:~/src/cyrus-imapd$ git log --oneline --since 2016-01-01 | wc -l
1683
brong@bat:~/src/cyrus-imapd$ git log --oneline | wc -l
14327

Looking at the code changes, there’s a ton of churn too:

brong@bat:~/src/cyrus-imapd$ git diff 4cc2d21 | diffstat | tail -1
876 files changed, 107374 insertions(+), 97808 deletions(-)

Which includes some really exciting things like redesiging the entire mbname_t structure to allow converting names between internal and external names really reliably and manipulation of mailboxes without any dotchar or hierarchy separator issues, which removes the cause of a ton of bugs with different configurations in the past.

In terms of new features, there is a full backup/restore system built on top of the replication protocol. There’s a fairly complete JMAP implementation. There’s much better fuzzy search support, built on the Xapian engine.

A large focus of our development this year has been making things simpler and more robust with APIs that hide complexity and manage memory more neatly, and this will continue with a lot more work on the message_t type next year. So there’s been plenty of improvement, not all of it visible in the headline feature department.

And it’s not just code. We’ve moved all our issue tracking to github and Nicola unified our documentation into the source code repositories, making it easier to contribute pull requests for docs.

Test all the things

As Chris mentioned in his post about our test infrastructure we’ve been increasing our test coverage and making sure that tests pass reliably. I’m particularly proud of the integration of ImapTest into our Cassandane test suite, and the fact that we now pass 100% of the tests (once I fixed a couple of bugs in ImapTest! The RFCs are unclear enough that Timo got it wrong, and he’s really reliable.) I also added support for CalDAVTester into Cassandane at CalConnect in Hong Kong this year.

Robert has added a ton of tests for all his JMAP, charset and Xapian work.

Our test coverage is still pretty poor by modern development standards, but for a >20 year old project, it’s not too shabby, and I’m really glad for Greg’s work back when he was at FastMail, and for the ongoing efforts of all the team to keep our tests up to date. It makes our software much better.

In particular, it makes me a lot more comfortable releasing new Cyrus updates to FastMail’s users, because for any bug report, the first thing I do now is add a test to Cassandane, so our coverage improves over time.

Going down the FastMail path

To build the 2.5 release, I sat in a pub in Pittsburgh with a giant printout of the 1000+ commits on the FastMail branch and selected which commits should go upstream and which were not really ready yet. The result was a piece of software which was not exactly what anyone had been running, and it kind of shows in some of the issues that have come out with 2.5. The DAV support was still experimental, and most of the new code had never been used outside FastMail.

After releasing 2.5, we looked at what was left of the FastMail specific stuff, and decided that the best bet was to just import it all into the upstream release, then revert the few things that were really single-provider specific and re-apply them as the fastmail branch. To this day, we have only between 10 and 50 small changes away from master in FastMail production on a day-to-day basis, meaning that everything we offer as the open source version has had real world usage.

So this means that things like conversations, Xapian FUZZY search (requires a custom patched version of Xapian for now, though we’re working on upstreaming our patches), JMAP (experimental support) and the backup protocol are all in the 3.0 betas. Plenty of that is non-standard IMAP, though we match standard IMAP where possible.

Version 3.0

There is both less and more than we expected in what will become version 3.0. The main reason for a new major version is that some defaults have changed. altnamespace and unixhierarchysep are now default on, to match the behaviour of most other IMAP servers in the world. We’ve also got a brand new unicode subsystem based on ICU, a close to finished JMAP implementation, Xapian fuzzy search, the new Backup system and of course a rock solid CalDAV/CardDAV server thanks to Ken’s excellent work.

Ellie released Cyrus IMAPd 3.0beta6 yesterday, and our plan is to do another beta at the start of January, then a release candidate on January 13th and a full release in February, absent showstoppers.

Plans for next year

Once 3.0 is out, we’ll be continuing to develop JMAP, supporting 2.5 and 3.0, and doing more tidying up.

As I mentioned in the Twoskip post, there are too many different database formats internally, and the locking and rollback on error across multiple databases is a mess. I plan to change everything to just one database per user plus one per host, plus spool and cache files. The database engine will have to be totally robust. I’m working on a new design called zeroskip which is going to be amazing, as soon as I have time for it.

I also plan to add back-pointers to twoskip (it requires changes to two record types and a version bump) which will allow MVCC-style lockless reads, even after a crash, and mean less contention for locks in everything. It’s all very exciting.

We’re heavily involved in standards, with JMAP in the process of being chartered with the IETF for standards track, and our work through CalConnect on a JSON API for calendar events. Cyrus will continue to implement standards and test suites.

The core team is Ken at CMU, Robert S consulting along with the FastMail staff: myself, Ellie, Nicola and Chris. Of these people, Ellie and Robert are focused entirely on Cyrus, and the rest of us share our duties. It’s been fantastic having those two who can single-mindedly focus on the project.

There’s plenty of space for more contributors in our team! Join us on Freenode #cyrus IRC or on the mailing lists and help us decide the direction of Cyrus for the future. The roadmap is largely driven by what FastMail wants because we’re paying for the bulk of the work that’s being done, but we’re also willing to invest our time in the community, supporting other users and building a well-rounded product, we just have to know what you need!