Improved spam filtering: avoiding email forwarding services

In a recent blog post, I mentioned that one of the ways to improve spam filtering is to avoid using a forwarding service, and especially if you have your own domain, to point the MX records for your domain directly to our servers (Enhanced or family/business accounts only).

The reason for this is that FastMail does a lot of work at the SMTP stage (when email is transferred from an external system to FastMail) to try and identify spam bots and block them while letting legitimate through. Unfortunately many of these processes can only work when the MX records for your domain point directly to our servers. If you point the MX records to another system, and forward from that system to FastMail, we can’t do the spam bot detection, and so more spam will get through.

To give you an idea of how effective this is, one of our users - the author, Simon Cann (http://simoncann.com/) - recently changed over the MX records for his Noise Sculpture domain (http://noisesculpture.com) to point to us instead of using a forwarding service. After doing so, he had the following to say:

As you suggested, I pointed the MX records for the Noise Sculpture
domain at FastMail and have been blown away by the results.  I’m now
getting maybe three or four pieces of spam in a day and the few pieces
of spam that are getting through are being correctly identified as
spam and put in the Junk Mail folder.

Simon went on to explain why he hadn’t made the change sooner.

I came to FastMail a year ago following an unexpected and highly
inconvenient melt down of my web host who was also by default my email
host for my simoncann.com domain.  This was embarrassing and
potentially lost me business (I was in the middle of pitching a book
proposal to a group of literary agents).

On signing up with FastMail I immediately pointed the simoncann.com MX
record at FastMail.  My other website for my music-related books
(Noise Sculpture) was hosted by a different web host and so I
forwarded all of my email from that domain to FastMail.  I didn’t see
a need to change the MX record–the web host was working well (still
is) and I couldn’t be bothered to start hacking the DNS records.

Long story short, as the simoncann.com and noisesculpture.com emails
were sent to a single FastMail account, I hadn’t realized how much
simoncann.com spam was getting killed by FastMail.  However, now that
I have changed the MX record for noisesculpture.com, I can see just
how much spam was getting through from that domain because I was using
forwarding rather than pointing the MX records.

This change has stunned me (in a very good way).  I’m so impressed
with the change (allied with your excellent service) that I’ve just
renewed my FastMail subscription to cover the next few years.

For the technically interested, the main things we do at the SMTP delivery stage to detect these spam bots are:

• XBL blocking. The XBL is a
  highly accurate block list that lists the IPs of machines with known
  trojans and proxies. Independent testing shows that the ZEN RBL (of
  which XBL is a part) has a high block rate, with basically no false
  positives
• Address enumeration detection. Spammers are always trying to
  find new addresses to send their spam to, and they do that by
  enumerating email addresses at hosts. We have a system that will
  automatically detect this behaviour, and block the spam bot from
  discovering any additional information about whether an address is
  valid or not
• Greylisting. Greylisting is a process designed to detect poorly
  written sending email servers, of which almost all spam bots are.
  While greylisting is very useful, it is sometimes frowned upon
  because poorly implemented versions would delay email delivery. The
  FastMail implementation uses a number of
  tweaks
  to ensure that legitimate email is delivered with no delay, while
  spam email is delayed or stopped from delivery
• RFC violations. When a server is transferring email from one
  system to another, there’s a series of conventions they have to
  follow defined in the SMTP RFC
  2821. Again, spam bots are
  often poorly implemented, or try and use tricks to work around this
  conventions. By detecting some of the common violations, more spam
  bots can be detected and blocked
• Custom spambot detection. Through our large and long term
  experience with running an email service, we’ve been able to closely
  analyse how spam bots try to deliver email to our servers. By using
  this data, we’re able to pre-emptively detect spam bots early only
  after they connect, and block them from delivering their spam email
  at all