Sharing my server horror story

After hearing all of the trials the folks over at SD had to go through, I thought I would share my own horror story.

*WARNING*
If you are a sys admin, net admin or anything related these pictures may seem a bit graphic.

It was a calm day in my world. I was just settling in for an afternoon of C&C. Our building has some sort of whisper effect. If you say it at one end of the hall you are bound to be heard at the other. So when I heard the sound of water, I just assumed it was the kitchen down the hall. Boy was I wrong.

The accounting staff is screaming. It was raining in their file room. Can you guess what is right next to the accounting file room? You got it, my servers. I open the door to two inches of water on the floor. Balls out; I rush in to start shutting off equipment. It is pouring in there. I am soaked and forget a graceful shutdown I am slamming power switches (I somehow thought that would be safer than pulling the cords out of the back of the servers.).

We call the building and they quickly determine that the water was coming from the A/C unit installed in the server room. How much water could an A/C make? Ours was fed by the same water system that the building was using for their A/C. So basically it was dumping hundreds of gallons of water a minute. The pressure was so strong that it cleaned all of the I-beams of their fire protection housing (3" thick).

Someone 14 years ago installed this A/C. They needed to cap one side of a T connector used in the installation. So they capped the copper pipe with a galvanized fitting. I am no metallurgist but I know water with chemicals and mixing of metals can really spell disaster. The cap popped off and the water ran free. (the cap was directly above my 42u server cabinet, which was right next to my PBX, which was right next to my switches)

Now everything in the room is wet, including me. All of the devices were powered down, but the UPS was still in the wall and still on. You really can't imagine this situation getting worse, but it does. The funny thing about batteries when they are connected together in mass and then you sprinkle a little water over them, they catch fire. Well they didn't catch fire, but they were arcing and caught all of the plastic parts of them on fire. So now the building maintenance people want the rack out of the building (we were on the 14th floor). I put the fire out as best as I can, reach over to the wall and grab this twist lock plug (110v 30amp) and pull it from the wall. Now I have to roll this into the service elevator and down to the loading dock.

Once I have it all downstairs, I have to begin removing the servers from the rack. There is nothing more depressing than picking up our main file server and having a gallon of water run out. We got it all back up stairs. Most of it was in good shape. All the resource cards in my PBX are covered with this chemically treated water. My servers all have some degree of the same water in them. We took everything apart, labeled it and left it over night to dry out. We couldn't do anything until it was dry.

The next morning I came in to find all of the water dried up, but this nice residue of a crystal powder. Salts and other chemicals were used in the treatment of the condenser water. Now all of this was in these servers. I wiped the PBX down and decided to power it up. It came up fine so I shut it down and moved it back to the rack.

The rest of the servers went to our outside support company. (we are a small staff so we use them for incremental support) These guys were great. They had new servers there that afternoon and they moved out disk arrays over to the new servers. Two days after we were flooded, we were back up with all critical services. We still had some issues, but we were back in business.

The statistics on a company’s survival after a catastrophic data loss are terrible. (Almost half don't make it a year) Fortunately, we lost NO data. Even more amazing is that we didn't even have any corrupt files in our COBOL accounting system. We have 20 accountants. I guess when the water came they stopped working.

This was a real trial. I think that first week and a half after I put in 124 hours. Now I didn't get in too much trouble because I risked my life, but our safety director (HSE for those that know) did fuss at me a bit for going in there so balls out. However, the company did cut me a check for all of my efforts and I did get a nice raise that year.

We made it through that trial and got a great story to tell. I'll never forget the call I made... "What's not under water is on fire, send me every free set of hands you have.

Now how about some more pics.

I hope I never have to go through that again.

BTW we moved a few months later and built out a whole new infrastructure that included redundant A/C units that were not located over the server room and an APC Symettra UPS with a remote kill switch that shuts the whole server room down at the touch of a button.

business

Locked Post

Advanced Search

Watch this post

Do not email me updates for this post Email me updates for this post