As a DBA, you have the opportunity to act like a firefighter. In fact the term ‘fire-fighting’ is very often used to describe those tasks needed to find the root cause of a system outage because there are a lot of similarities. Â When fighting a real fire there is more to it than the obvious need to rescue victims, Â you have to hunt for the cause before you can decide how to treat it. Sometimes water helps; sometimes, as with electrical fires, water makes it worse. Iteratively you work through the causes and treatments so that the fire is extinguished as quickly as possible without destroying more property and lives.
As a DBA, when we are fire-fighting, there is almost always some major catastrophe at the surface. The corporate database server is down that the customer website is based on, and there is no backup. Without this database working, everything is dead in the water. So one works tirelessly towards a solution, throwing all caution (and paperwork) to the wind and getting the revenue flow going again. This sort of bravery to face the unknown server gremlins impresses pretty much everyone you work with, especially the people who sign checks. Why, then, even worry about spending your days monitoring servers, backing up databases, and concerning yourself about data integrity at all. Â Having the bravery to work through a problem in a crisis is more rewarding.
However, the truth is that DBAs do their best work while wearing their underpants inside of, rather than outside of, their trousers. Curiously, this is true of firefighters too. If you ask any sane firefighter what the best part of their job is, expecting heroic tales of rescuing young ladies from burning bedrooms, they may surprise you by recalling the time they found a problem that prevented a building from burning in the first place. Though they all like driving that big red truck at speed while waking up your sleeping baby with the siren, they much prefer fire prevention to fire-fighting. Â In fact, the most important person in the hierarchy of fire station is the fire marshal. Their job is to come into a building, inspect it, and make sure that all of the necessary precautions are in place to prevent a future disaster.
As a DBA, your job is very similar in some ways to that of the fire marshal as you need to help make sure servers that are built have all of the necessary precautions built in to be fire proof. And sometimes, just as builders and architects may dislike the fire marshal, so too stands the DBA as that annoying voice reminding you to get the redundant servers, disks, well, pretty much everything. It can take a dose of bravery to make sure that, when a database solution is stood up, that it is done the right way. Building a computer system for a database server is not the same as building a general file server, and never has been. To make matters worse, the guidelines for building a database server change constantly as technology advances. For example, the advent of solid state drives means that data access is less of an issue than it has ever been, but building a server is still not as simple as tossing in a few solid-state drives and imagining that it solves all problems.
Truly noble geek-bravery is shown when you get to stand up in front of your company’s president and explain why you want to spend all that extra Â money for disks, tapes, offsite storage, etc., making the price tag way more than the server solution than your coworker specified. Nerd-Heroism is in explaining how essential it is to the organization that you make sure your server is able to handle exceptionally large user loads, is able to handle any small disaster like a simple drive failure, and finally is able to deal with a large disaster like the building where it is housed burning down. The extra money may never actually be necessary, but no matter how much you hope for it, if the fire arrives, that firefighter isn’t going to run into your burning office to rescue your server…