Massive increase in email connections to our servers

About 4 days ago or so, something went crazy with the spam zombie machines out there. Previously the spam sending software spammers were using was acting like reasonably well behaved email sending software. It would connect to us, trying to send it’s spam, then disconnect just like any other email sending system on the Internet. They’d do that every now and then, maybe an hour or two between attempts. Still, with 100,000’s of machines, that’s millions of attempts a day to send spam.

Now however, the zombie machines and software have just gone insane and are connecting over and over every few minutes, but mostly doing nothing during the connection. While that in theory might seem fine since they do nothing, it’s not. When you have 200,000+ machines connecting to you every few minutes, even if they do nothing you still have the connections to deal with, the RBL DNS lookups, the rate limiting lookups, etc.

The result was a significant jump in load on the incoming servers, significantly above the load jump we’ve seen over the last couple of months even.

To combat this, we’ve had to invoke some old code from a previous “bombing” attempt we had a while back. This code continuously scans the logs looking for particular aberrant behavior and then put those IPs on a special “early” block list which means as soon as the machine connects, it’s sent a response of:

454 Service temporarily unavailable; Client host [x.y.z.a] blocked
using internal list; Access denied

And disconnected. Over the course of a couple of hours and days (as infected computers out there were turned on and off), we’ve built up a list of over 200,000 IPs that are now being “early blocked” like this. To give you an idea of how big the surge is, almost 3/4’s of all connections are now being “early blocked” by this list. That means incoming connections have probably almost tripled in in the last 4 days.

This is also something we can just confirm by the size of our log files. Normally our email processing files are rotated each day, but we’re now having to rotate them multiple times a day because they’re reaching their limit of 2 gigabytes in size!

Our only current worry is that somehow we’ve blocked some other services incorrectly. We’ve had one report from a user who’s scanner has been blocked (it’s an internet enabled scanner that you can setup to email you when you scan something, unfortunately it seems to be designed for LAN networks, and polls the SMTP server you’ve setup every 60 seconds to see if it’s alive, much like the spam zombies are *sigh*).

Some more information about the current spam wave that’s going on is at Extreme Tech.

Update: It seems some badly run sites were being blocked. Some sites with incorrect DNS setup were being identified as “dialup/dsl” machines. Some other sites seemed to be doing the same signature of the spam zombies, namely “connect, do nothing, disconnect”. Some other sites were sending rapidly to many unknown recipients, also a sign of a spam zombie trying to enumerate usernames. We’ve tightened up the blocking criteria some more, removed a number of existing blocks, and put some common hosts on an IP whitelist so they’re not blocked again in the future.