15 Years of Pobox: The day of the generator
Post categories
This article was originally published as part of the Pobox blog. Pobox was acquired by Fastmail in 2015.
Can you believe that Pobox has been around for 15 years already? It seems like just yesterday the website was only one or two pages, and we were still trying to figure out how to keep track of the checks people were mailing us.
How quickly times changed! And yet, there were still plenty of ups and downs ahead. In commemoration of 15 years of Pobox, I asked current and past staffers if there were any stories they’d particularly like to see on the blog. Mark asked to hear the story of the water main break, after he found a CNet article on it.
On Friday customers of POBox.com, an email forwarding service, were baffled by an outage in what they say has otherwise been a highly reliable service. It turned out that the eight-hour shutdown–in which no email was lost–was not caused by a faulty router or programming error, but by the kind of a common building problem that faces “brick and mortar” companies every day: a water main break.
I asked Helen Horstmann, our CEO, if she would tell us about that day, way back in September 1998.
I refer to it as “the day of the generator.” Back then, we had a T1 in the office, and virtually all of the important servers were there. I walked up to the building, where a small crowd of people was hanging around, waiting to be told they could go in. A water main had broken under the street. The basement was flooded. The elevators weren’t running, because the power was out. WAIT. “The power is out?!”
We waited for about 30 minutes before panicking. Maybe it was a little water main break. Maybe we everything would be fine in just a few minutes. But more and more PECO workers kept showing up, and finally, one of them announced, “You all might as well go home – the power’s going to be out all day.”
I was floored. I ran over to him, and said, “All day? What does that mean? Like, 3 or 4 hours?” He said, “Maybe. Maybe 12 hours. It will definitely be fixed by tomorrow morning, though.”
I asked if the building was still open, and the desk guy told us we were welcome to walk up, but the building’s emergency generator only powered the Exit signs, so the stairwell would not be lit. Using our cell phones as flashlights, we hiked up the nine flights of stairs to the office, where our battery backups were beeping madly. Two people started shutting down the non-critical servers, to reserve as much power as possible for the important ones.
We needed a plan, and fast. We needed our own emergency generator.
I asked if it was possible, any way, any how, name your price, to piggyback off the building generator. But they hadn’t had an outage in years, and hadn’t been keeping the tank full of gasoline. They figured they had no more than 6 hours of power, and they needed it all for the emergency workers.
The building had sealed windows, so running a generator in the office was also a non-starter… but we had to have power! What if we were down for 12 hours? What if we were down LONGER?!
I found an equipment rental company 30 minutes away that had a generator they were willing to rent for 24 hours. We were all early twentysomething city dwellers, so the only staffer with access to a vehicle was a customer service agent who was also in a band. He and I headed over to his apartment to grab the band van. No drums to schlep today – we’re schlepping a generator!
In the meantime, someone else was getting access to the roof, and estimating the distance from the roof to our office with ethernet cable. Even after we got the generator, we were going to need a 47 foot, 220 volt extension cord (or 15 meters, for our metric friends).
Luckily, the generator guy referred us to an electrical supply company that was on our way back. The cable was on a 3 foot wooden spool (which we later used as a stand-in coffee table, in commemoration of the event). We rolled it out of the store and into the band van.
When we got back to the office, the desk guy took pity on us, and used a fireman’s key to turn on the elevator – I don’t even want to think about having to carry those things up 15 flights of stairs! We carried it the last flight to the roof, and started unspooling the power cable.
Did I mention the windows of that office didn’t open? We had to drill a hole in the window frame to get the power cable in. Thankfully, we realized that before we attached the plug head to the cable – drilling the first hole took almost 20 minutes; making it twice as big could have eaten up nearly an hour.
By now, the batteries had long since died, and all the machines were off. It was quieter than the office had been for years. We wired up the power cable, plugged in our main UPS, called up to have the guys on the roof start the generator and …
Nothing happened.
The battery wasn’t charging. We checked everything we could think of. The power inversion on the generator. The cable. The plugs. The UPS. Nothing. Nothing worked. This may have been the point where I went into the stairwell and shouted obscenities.
We started calling electricians. One after another told us they couldn’t come until tomorrow. (At this point, it was nearly 3 in the afternoon.) We started calling friends who knew anything about electricity. We knew we were at the end of the line.
How many times did I walk up and down those 9 flights of stairs? How many times did I bug the PECO guys for a new estimate? I know someone tore me away from the building at some point to eat. There was nothing left to do but hope that the power would be back on shortly.
Around 6 PM, we were sitting on the floor when the overhead lights flickered on. Everyone jumped up and started turning computers on. Within 20 minutes, everything and everyone was running – watching logs to make sure everything was moving as quickly as possible, answering the piles of customer service mail that was coming in.
That was back in the day when Pobox cost $15/year. So, if you’re wondering what that extra $5/year is buying you, well, you could say that it’s a low cost for a lot of extra peace of mind. :)
Nowadays, of course, Pobox has redundant datacenters in world-class facilities, running on enterprise-grade hardware. Things can still go wrong, but now we’re counting on our vendors and partners to solve the problem, not the local electrical supply store. Thanks for sharing that story, Helen!
Do you have a story to share about Pobox? Drop me a line at [nessie@pobox.com](mailto: nessie@pobox.com), and it just might end up on the blog!