New POP/IMAP server version

Over the last 24 hours, we’ve rolled out a new POP/IMAP server version for all users. This new server is the result of months of great work by Bron and includes many improvements and fixes. Not that many of the fixes are currently user visible changes, but they are significant internal improvements that help improve reliability, conformance and performance, and will allow us to build some future features we’re looking at.

• Email replication improvements

  Email replication has been made much more efficient and reliable.
  The format includes
  CRC
  auto-integrity checking features, so that any unexpected mismatches
  between both ends are automatically detected and fixed. It can also
  recover automatically from unclean shutdowns or machine crashes
  where “split brain” has occurred, automatically fixing up mailboxes
  and messages. The format has also been made future extensible,
  allowing more features to be added without compatibility problems.

• Performance and integrity improvements

  The internal mailbox format used to store emails has been
  significantly reworked. The new format has reliable locking
  semantics to remove all race
  conditions. It also
  stores and checks
  CRCs on all
  record data and cache data, and SHA1 checks on all message files.
  This ensures that any corruption in any data is detected early and
  can be dealt with. By moving around some of the data (such as the
  user seen state), and only lazily opening files as needed, the new
  format also improves performance in many common cases.

• Strict MODSEQ, QRESYNC support and full IMAP test suite conformance

  Recent extensions to IMAP allow clients to more quickly synchronise
  data between the server and the client (eg.
  CONDSTORE/MODSEQ and
  QRESYNC). While the
  server has supported CONDSTORE/MODSEQ for a while, unfortunately it
  was a bit buggy in some situations, causing message seen state to
  get out of sync. The server now correctly and accurately support
  CONDSTORE/MODSEQ, and also supports the current QRESYNC standard
  that will allow clients that support it to sync even faster. We also
  now correctly pass detailed IMAP stress tests.

• Major code cleanups

  All of these improvements have also been done with major internal
  code cleanups. This will allow us to continue building additional
  functionality and features more easily in the future, and to more
  easily fix and debug any other issues that are encountered.

Unfortunately no good deed goes unpunished, and even though we’ve been testing this code ourselves and on a sub-set of users for weeks with continuous improvements, unfortunately some bugs did get through when we finally rolled out to all users. Then in the attempt to fix these issues as quickly as possible, we also introduced some other issues. The net result was that for about 12 hours, there was a sequence of small but potentially annoying bugs that would have affected different sets of users.

• On first access, we upgrade a mailbox to the new format. During the
  upgrade, we found some existing caches had allowed invalid data to
  enter them, causing corruptions on upgrading which caused problems
  when accessing these mailboxes. These cases are now caught and new
  cache data is built from the underlying message files
• While reconstructing the mailboxes that had been incorrectly
  upgraded by the above code, a quota error caused some peoples quota
  to temporarily be double their actual used amount. This has been
  fixed now. If this bug sent a user over quota temporarily, it
  shouldn’t be a problem. When a user is over quota, we return a
  temporary 4xx error, which means no messages should have been lost,
  the other side should just have re-delivered when they were back
  under quota.
• IMAP IDLE wasn’t returning new messages, only updating existing
  messages, causing pushing of new messages to most email clients to
  not work
• Mail App has a bug with parsing IMAP IDLE unsolicited fetch
  responses that contain more than flags information. We’ve added a
  workaround for this Mail App bug
• The IMAP COPYUID response was producing a non-conformant result,
  which caused some programs to report an error (Outlook 2010)
• POP3 was using an optimised mode if a mailbox was empty.
  Unfortunately the code to mark a mailbox as “non empty” wasn’t
  working properly when messages were delivered, but was working for
  IMAP logins. This meant that messages delivered wouldn’t be
  downloaded by POP until you did an IMAP or web login
• The POP3 TOP command wasn’t working, causing some programs (Outlook
  in POP mode) that download email headers to fail
• The POP3 UIDL command with a message ID was producing a
  non-conformant result, which was parsed incorrectly by some
  programs. This caused some POP programs to download the same message
  more than once, or to delete off the server before it should have
• Update: An update to UID sequence handling  caused the mailbox
  status command to report unread messages as read and vice-versa,
  causing the unread count on folders to actually be the read count
  for a short while.
• Update: The XLIST extension wasn’t working. This has been added
  back, so client that support it will automatically pick the right
  Sent Items, Drafts, Trash, Junk Mail folders when setting up a new
  account
• Update: NOOP on Mac Mail. Like the bug above with Mac Mail and
  IDLE, this was affecting the NOOP command as well
• Update: Storing the \Seen flag + another flag on a message that
  already had the \Seen flag would cause \Seen to actually
  disappear. This mostly manifested as when deleting a message, it
  would cause it to become marked as “unread” again

All these issues have now been fixed, and we’re closely monitoring all the server logs to see if there’s any other issues, but at this stage we believe that the new server and code is working correctly for all cases we’re aware of and for all clients, IMAP and POP.

All this new code is part of the open source project cyrus, and we’ll be pushing this code back to the main cyrus code base, which will eventually form the basis for a new cyrus version 2.4. For those interested in technical details, Bron will post to the cyrus mailing lists when he’s had a bit of time to compile all the documentation and technical details.