As some of you actually noticed, the blog was down for a few hours today. I have monitors that continuously watch the Internet connectivity for my server, and today around noon every possible red light and warning buzzer went off. With a little bit of diagnostic work I figured out that it wasn’t one of my servers — it was the entire Internet connection to my server closet, dead as the proverbial doornail.
My servers are not in the boondocks with me — they’re located in the basement of a friend’s home in La Mesa, which is a relatively civilized place, with cable, DSL, grocery stores, etc. My friend’s home is about a 40 minute drive from my home, so whenever something like this happens (thankfully not very often!), I’m faced with the need to make a roughly 90 minute round trip. Usually the problem is simple to fix: just reboot (or power cycle) the DSL router, and I’m back on the air.
But that didn’t work today. Something more serious was wrong. And that meant I needed to get a cell phone (for nobody was home at my friend’s house), and the contact information, circuit number, etc. so I could call Covad (my ISP) and troubleshoot it with them. So off I went, back to my house to pick up this stuff, and then back again to my friend’s house.
When I got back to my friend’s house the second time, my friend (also Tom) was home, and he joined me for the troubleshooting. I went through all the troubleshooting procedures with the very friendly Covad folks, but they couldn’t bring the connection back to life. Through our testing, they figured out that there was a “connectivity problem” — in other words, that somewhere between the telephone company’s office and my friend’s house, the wiring was broken. Fixing this entails a long process involving phone company representatives, appointments, etc. It would take days or (worst case) weeks to get my system back on the air. It didn’t look good.
As a last resort, I decided to check the box where the phone company’s wiring connects to the wiring I had installed. I opened it up, made sure the connections were tight — it all looked fine. But my friend noticed that a foot or so from the box, the wiring to my server cabinet was taped together. The tape looked old, but neither of us could remember having made a junction — especially one that was taped like that. I figured the most likely thing was that we’d forgotten doing it — after all, it had been over a year since we installed this thing. But just for the sake of completeness, without any real hope that this would prove to be the problem, I decided to unwrap the tape and check it out.
I’m very glad I did, as what we discovered was, in fact, the problem!
The first thing we noticed under the tape was that all eight wires in each cable were twisted together — with the insulation still intact. That certainly wasn’t going to work! And after I untwisted them, the two pieces of cable just fell apart — it most definitely wasn’t connected at all.
Ah ha! The problem! And easily fixed, too — my friend noted that there was enough slack that we could simply connect the fresh cut directly to the phone company box, and voila! Everything came back up and worked great.
Which left one mystery: just how did the cable come to be cut at around noon today? My friend then remembered a crucial little bit of information: his gardener had been there today, and he had been wielding a power trimmer. This is just the sort of tool that could cause the damage we saw, and the timing was right as well. Apparently the gardener whacked the cable (sawing off my server cabinet from the Internet) and either decided to hide the accident by taping the wires back together, or he was so ignorant of the mysteries of electronics that he thought he actually fixed it by twisting it together and taping it up. He never mentioned it to my friend.
That’s a new one on me — my “data center” taken out by a gardener!
No comments:
Post a Comment