Av rating:
Total votes: 53
Total comments: 14


Phil Factor
The Data Center that Exploded
28 October 2008

A while back, in a Simple-Talk editorial meeting, someone bet Phil that he couldn't come up with a Halloween story. To our surprise he said he could, as long as he didn't have to keep to the strict literal truth. In the end, he came up with a story about a story, and it is true that he first told the story in a data Centre at Halloween!

A Halloween Tale

“There’s always a good reason why Database Applications go wrong. It is always down to human error or mechanical failure.”

The older I get, the more I feel compelled to contradict confident assertions.

“You might think that”, I replied, “but you probably don’t remember that time back in 1992 when the overnight clearing system for the Imperial Bank crashed, and couldn’t be restarted for twelve hours.”

“No”, he said firmly.

I wasn’t discouraged.

A Prank is Plotted

Have you ever had a garrulous colleague at work? If you have a pig-pen you can escape; but in an open office area it is torture. Alistair hadn’t enough work to occupy his day. Out of work hours, he lived alone; at work, he chattered unceasingly. To make things worse, Alistair was cocky. He was a contractor, and like many of his breed, he spent too much of his energies telling us how clever he was. When you are working ten hours a day in close proximity to a guy like this, it can get wearying. To make matters worse, he was too prone to playing ‘practical jokes’ on people, and there was more than a whiff of malice about them.  I loved to escape occasionally to the Data Centre to call in on ‘Bill the Minder’, the manager in charge.

 It is hard to describe the sheer immensity of the data centre of the Imperial Bank at that time. There was floor upon floor of expensive steel cabinets, weird robotic tape systems, and curious islands of obsolescent hardware, all cast in subdued lighting against cream-painted concrete.  High up on a mezzanine floor shone the windows of the operators offices. There, in a strange middle-earth grotto lived a typical breed of operator; opinionated, slightly belligerent, but single-mindedly devoted to the health of the systems in their care. It was there that Bill had his office and ruled his kingdom. For me, it was paradise; Technology-play. Bill and his team could check and control everything. Computers, Lights, air conditioning, the power to the systems: Screens were set up to monitor everything that SNMP could reach.

Out in the cavernous hall of the old data centre, there were countless indicators for alerts and activity, so that wherever you were, you could be aware of a sudden problem. In the semi-darkness of the windowless building, one could imagine being in a strange satanic cathedral.

It was while I was standing on the tiled floor, staring up at the lights of the office, that the idea came to me of a Halloween prank. I thought that, maybe, it would earn us all some peace from Alistair’s prattling.

I went up the stairs to see Bill. ‘Would you be able to set up a bank of disk lights and make them go off when I give a signal?’ He looked contemptuous. ‘We already have them, we set some up for the last visit of one of the directors of the bank. They were worried that there wasn’t enough so see so we lashed them up.’  I told Bill of my plan for Alistair. He agreed to do everything he could.

Bugbears

“Y’know”, I started “how the word ‘Bug’ came about don’t you?” I asked Alistair innocently. He and I were working away in the third-floor switch room where the test database was kept.

“Yes, everyone knows that. An insect was found in a computer. There was a fault and a moth had to be extracted from between the contacts of a relay. They wrote in the log that it was the first actual time that a computer had been ‘debugged’. They stuck the moth in there. That was where the term originated”.

“Not at all. They were just amused at the time because they all used the word debugging already, and they wanted to record the first time that a real bug caused the damage. In fact, Radar Electronics used the term during WW2. And before then, any mysterious problems in machinery were called ‘bugbears’.  The Welsh referred to them as ‘Byg’.

“Oh yes?” Alistair sneered. One of Alistair’s tireless themes was his militant scepticism. He railed against all superstition.

“It is an interesting word. You can track it back hundreds of years. People suspected that any machinery, even a cart, could be infected by Bugbears, which were a sort of mischievous spirit like a gremlin. ‘Hobgoblins and buggybeares’. Even the millers in the old watermills would talk about ‘buggybows’ getting into the mill works. We have references to the word in the oldest printed books in English.”

“Nah. It’s an American word. I think Thomas Edison used it.”

“Well, so did a scholar in the twelfth century, writing about devils and spirits: There was a Celtic god, Bugibus, who was associated with avenging or malicious spirits.”

“Phil, you make up the most incredible rot, you really do.”

I shrugged. “Look it up yourself, it is all in the Oxford English dictionary.”

He looked doubtful, which was a shame as I could have won some money on a bet.

“I’ve been doing some work on the subject myself at the Harry Price Library at the Senate House.” I added. “The custodian ferreted out several boxes of miscellaneous papers, mainly bits of old books  from Powys Castle library that had never really been sorted since their bindings fell apart. On one of them was a poem; an  ‘Elegy to Buegibos’.  It was written on what seems to have been the flyleaf of a fifteenth century printed book. I photocopied it and had it transcribed, and partly translated, by a friend at the British library. Buegibos seems to have been that vengeful Celtic spirit. “

“OK.”, interjected Alistair, challengingly, “what did the poem say?”

“It starts off  ‘Darkest Dark, and deepest deep, Bugbears rise from deaths last sleep, Come, with the funeral’s tolling bell, from vampire’s lair and shrieking hell…’”

At that moment, the lights went out. It is a simple trick, which is done by leaning against the wall and tapping the switch with your shoulder-blade.  ‘Oh blast, the condenser in the overhead lights is a bit dodgy, I must get it fixed.”. I lit a cigarette lighter, and the room became suffused by an eerie glow. “I’ll try switching on and off again”. Before he had time to look, I had the lights on again.

“All rot, this is”, spoke Alistair, slightly hotly, I thought.

“It could be.” I shrugged. ‘I’m a historian, not a spook-chaser. The poem is genuine enough. Let’s try it out in the data centre and see if it really does infect big computers.’

“You must be crazy! We’d be sued for millions if we caused damage there”

“By reading an old translated poem in a server room? What would we be convicted of? Witchcraft?  It would be simple, anyway, because ‘Bill the Minder’ isn’t around after seven in the evening and the production team are watching ‘Big Brother’ on one of the monitors then. I’m allowed access to the server room, and I have a swipe.”

Alistair looked doubtful. “Worried by a silly poem now are we?” I commented, tantalisingly, “not quite certain are we?”

There was silence while he balanced the pros and cons.  He was imagining his humiliation if I was able to tell the office that I'd tricked him into believing that it was possible to cause a computer failure by reciting a poem. The pros proved weightier. “Oh, all right, if you want to make a fool of yourself, I’ll be there to witness it, and maybe pass on the story to the IT department.”

Oncely, Twicely, Cast the Runes...

It was Halloween. As rare as snow at Christmas, there was a thunder storm during Halloween. It was a perfect adjunct to the surprise I planned to get the wind up Alistair. Alistair followed me into the dimly lit data centre.  We gazed all around at that huge cathedral to technology, clicking sighing and flashing in its restless routine activity. This would be perfect theatre. The office where Bill the Minder and his team lived and worked was in semi-darkness, lit only by flickering screens. Somewhere in the room was Bill, waiting to animate my little Halloween pantomime.

I got the script out, and cleared my throat.

  “Darkest Dark, and Deepest Deep, Bugbears rise from Death’s last sleep” I intoned.  The poem was vile, and deeply obscure. The mediaeval mind was full of ‘memento mori’ details that are not for the squeamish. Then it got down to business. I finally came to the crescendo, reading the poem aloud in a melodramatic voice, so that it echoed around the hall;

“….

By the restless ghouls that haunt by night,

While Buegibos laughs with a grim delight

Leave your torture and your pain

Taunt those living souls again

Let lightening flash, let thunder roar

Let the clouds sweat gobs of gore

Come through the night, come creep, come crawl,

To Bugbears wail and vampire’s squall,

Oncely twicely cast the runes,

Tear the wind with ghastly tunes,

Strip the apple from the Yew

Drip the venom in the brew,

Arise Buegibos spirit of the night

Infest with Bygs in this, thy ageless rite.”

 I shouted out that last sentence in a loud, declamatory voice. It was a prearranged signal to Bill the Minder. To my great excitement and delight, the banks of LEDs went out. Beautiful! Alistair bent forward in surprise, as if punched in the stomach. The next moment there was an immensely loud bang high up in the building, and a shower of sparks rained down. There were strange snapping and popping sounds all around. I was amazed. Bill had surpassed himself with this. Then I became aware of an eerie silence. The huge background hissing, humming, sighing and clicking had suddenly diminished. The computers were falling silent. Alistair gave out a screech.

 Bill rushed in through the entrance, looking wet and dishevelled.  ‘Hmm. That was a neat trick’ I thought.

“What the hell is going on?” he shouted; his face white. A good actor Bill was.  Alistair made another squeak and ran for the door. I gave a chuckle.

“Oh, yes, Bill, you surpassed yourself there. That was magnificent. How did you do that extraordinary explosion?”

Bill looked at me oddly

“I was stuck in a tube train for twenty minutes between stations. You were the only two people here. My pager went off just as I was walking up to the Data Centre through the rain.”

 ------||------

“Oh come on, Phil: Another of your fine stories. We know about that incident. There was a huge direct lightening strike on the Data Centre at the Imperial Bank that almost melted the steel frame and blew just about every piece of electronic equipment in the place. It was famous, it is in all the text-books, and led to the practice of twinning data centres at two different geographical locations.”

“Fine”, I countered, getting off my bench in the Data Centre where I’d just told the story, “What really happened remained a secret between Alistair, Bill and I. If you’re entirely sure that I’ve made the story up, then I just happen to have remembered the entire poem, and I’m going to recite it here and now. ‘Darkest Dark and Deepest Deep, Bu……’”. The rest of the poem was lost as they wrestled me to the ground.



This article has been viewed 7395 times.
Phil Factor

Author profile: Phil Factor

Phil Factor (real name withheld to protect the guilty), aka Database Mole, has 25 years of experience with database-intensive applications. Despite having once been shouted at by a furious Bill Gates at an exhibition in the early 1980s, he has remained resolutely anonymous throughout his career. See also :

Search for other articles by Phil Factor

Rate this article:   Avg rating: from a total of 53 votes.


Poor

OK

Good

Great

Must read
 
Have Your Say
Do you have an opinion on this article? Then add your comment below:
You must be logged in to post to this forum

Click here to log in.


Subject: Byg, Bugibus, bugaboo and bugbears.
Posted by: Phil Factor (view profile)
Posted on: Tuesday, October 28, 2008 at 12:22 PM
Message: See my blog post at http://www.simple-talk.com/community/blogs/philfactor/archive/2007/06/01/30276.aspx for the story of why we call them 'bugs'. Here, I slightly embroidered the tale to make it scarier. Artistic license if you must know!

Subject: Excellent story!
Posted by: imassi (view profile)
Posted on: Tuesday, October 28, 2008 at 12:39 PM
Message: I rather enjoyed that story, it gave me chills! Since you have lived to tell about it, I take it you were burned at the stake for witchcraft. Did they let you go when the scales showed you weighed more than a duck?

Subject: Drip the venom in the brew
Posted by: Corwin (not signed in)
Posted on: Tuesday, October 28, 2008 at 2:17 PM
Message: Bravo! Now, if you'll excuse me, I need to visit the data center.

Subject: Re: Drip the venom in the Brew
Posted by: Phil Factor (view profile)
Posted on: Tuesday, October 28, 2008 at 2:27 PM
Message: I deliberately left out most of the start of the incantation. it was too disgusting and could cause havoc in the wrong hands.

Subject: Nice one!
Posted by: Adam Machanic (view profile)
Posted on: Wednesday, October 29, 2008 at 8:55 AM
Message: Another great Phil Factor tale...

Subject: Excellent
Posted by: preet (not signed in)
Posted on: Wednesday, October 29, 2008 at 2:12 PM
Message: .

Subject: Great story!
Posted by: Petter (not signed in)
Posted on: Thursday, October 30, 2008 at 2:19 AM
Message: Thanks a lot!

Subject: keep it coming
Posted by: Dave (view profile)
Posted on: Thursday, October 30, 2008 at 7:23 AM
Message: Without your thespian wit Phil my small part in the idiocracy would be immeasurable less palatable. May they forever subsidise your musings.

Subject: Good
Posted by: Venkatesan Prabu (not signed in)
Posted on: Friday, October 31, 2008 at 1:46 AM
Message: Really an interesting article

Regards,
Venkatesan Prabu .J
Kindly project me for some other opportunities.

Subject: Really an interesting article
Posted by: Venkatesan Prabu (not signed in)
Posted on: Friday, October 31, 2008 at 1:47 AM
Message:
Really an interesting article

Regards,
Venkatesan Prabu .Jhttp://venkattechnicalblog.blogspot.com/

Subject: Poof
Posted by: Bobw111 (not signed in)
Posted on: Friday, October 31, 2008 at 10:54 AM
Message: When I went through USAF programming school at Keesler AFB in 1978 I had the nickname of "Poof".

The datacenter was in an old hanger.

The first time we were allowed to actually sign on to live terminals (Teletype) I put in my user id and password.

I tried about four times and couldn't get it to work. The instructor came over and explained what I was doing wrong.

By this time the whole class had stopped what they were doing and were watching me.

Just as I hit the carriage return (Enter) the lights went out in the room we were in.

And the rest of the building including the Honeywell mainframe datacenter...

The our instructor and several others actually had me and the rest of the class convinced for a while that the only cause they could find was my logging in.

Subject: Heh
Posted by: SQLWayne (view profile)
Posted on: Monday, November 03, 2008 at 10:54 AM
Message: If there's one thing that I've learned about Phil Factor from reading his posts, NEVER challenge him to a story. He has enough industry experience and broad knowledge that he could probably spin one for any occasion.

I'm personally waiting for one for Guy Fawkes Day involving, perhaps, networking.


And Bob, I have a similar story. Years ago I was taking a class on a Cromemco system, good old Hazeltine terminals if I recall correctly. Whenever I exited the editor, the entire system crashed and had to be rebooted. It got to the point that I had to tell the instructor when I was quitting the editor so that he could warn the others to save their work.

Subject: Again ..... a Gem
Posted by: Mal (view profile)
Posted on: Wednesday, November 12, 2008 at 7:48 PM
Message: Ummm I'm not invoking the Wyrds. I believe !

Well done Phil.

Subject: Anonymous commenting disabled
Posted by: Chris Massey (view profile)
Posted on: Thursday, November 13, 2008 at 11:48 AM
Message: I'm afraid anonymous commenting has had to be disabled in this article, due to spamming. If you want to leave a comment, you'll have to sign in or sign up. Sorry for the inconvenience.

 


















Raw Materials: Robotic Resources
 Derek is judged by a jury of his peers. Read more...

Raw Materials: The Trojan Horse
 Excuse me, was that geek or Greek? Read more...

Raw Materials: The Temptations of a DBA
 Derek hears voices. Read more...

Alan Kay: Geek of the Week
 The development of Object-oriented programming, the windowing User-interface, Ethernet and the Laptop... Read more...

Simon Sabin Says SQLBits
 SQLBits is the largest SQL Server conference in Europe. Because it is held on a Saturday, and is free,... Read more...

Linus Torvalds, Geek of the Week
 Linus Torvalds is remarkable, not only for being the technical genius who wrote Linux, but for then... Read more...

Driving up software quality - the role of the tester
 Have you ever wondered what a software tester does? Helen Joyce, test engineer at Red Gate software... Read more...

Coming Out as a Cancer Survivor - A Guide for Software Developers
 A personal perspective on the responsibilities of a cancer-surviving software developer Read more...

The Computer that Swore
 Database Developers occasionally get crazy ideas into their heads. Phil Factor should know; He... Read more...

Bad CaRMa
 From hope and euphoria, to desperation, firings and the ultimate demise of a company. Tim Gorman charts... Read more...

Over 150,000 Microsoft professionals subscribe to the Simple-Talk technical journal. Join today, it's fast, simple, free and secure.

Join Simple Talk