In my previous post, I wrote about becoming an Accidental DBA whether or not you had that title formally. I described the things a Minimalist DBA should focus on before jumping into performance tuning or renaming that table with the horribly long name (RETAIL_TRANSACTION_LINE_ITEM_MODIFIER_EVENT_REASON I <3 you.) In today's post I want to cover how you, personally, should go about prioritizing your work as a new Accidental DBA. You
Most accidental DBAS perform firefighter-like roles: find the fire, put it out, rush off to the next fire and try to fight it as well. Often without the tools and help they need to prevent fires. Firefighting jobs are tough and exhausting. Even in IT. But they never allocate time to prevent fires, to maintain their shiny red fire trucks, or to practice sliding down that fire pole.
How to Prioritize your Accidental DBA Work
- Establish a good rule of thumb on how decisions are going to be made. On a recent project of mine, due to business priorities and the unique type of business, we settled on Customer retention, legal and application flexibility as our priorities. Keep our customers, keep our CIO out of jail, and keep in business. Those may sound very generic, but I've worked in businesses where customer retention was not a number one priority. In this business, which was membership and subscription based, we could not afford to lose customers over system issues. Legal was there to keep our CIO and CEO out of jail (that's what ROI stands for: Risk of Incarceration). Application flexibility was third because the whole reason for the project was to enable business innovation to save the company.
Once you have these business priorities, you can make technical and architectural decisions in that context. Customer retention sounds like a customer service issue, but it's a technical one as well. If the system is down, customers can not be customers. If their data is wrong, they can't be customers. If their data is lost, they can't be customers. And so on. Every decision we made first reflected back to those priorities.
- Prioritize the databases and systems. Sure, all systems are important. But they have a priority based on business needs. Your core selling systems, whatever they might be, are usually very high priority. As are things like payroll and accounting. But maybe that system that keeps track of whether employees want to receive a free processed meat ham or a chunk of processed cheese over the holidays isn't that high on the list. This list should already exist, at least in someone's head. There might even be an auditor's report that says if System ABC security and reliability issues aren't fixed, someone is going to go to jail. So I've heard. And experienced.
- Automate away the pain…and the stupid. The best way to help honor those priorities is to automate all the things. In most cases, when an organization doesn't have experienced or dedicated DBAs, their data processes are mostly manual, mostly reactive, and mostly painful. This is the effect of not having enough knowledge or time to develop, test, and deploy enterprise-class tools and scripts. I understand that this is the most difficult set of tasks to put at a higher priority if all the databases are burning down around you. Yes, you must fight the fires, but you you must put a priority on fire reductions. Otherwise you'll just be fighting bigger and more painful fires.
Recovery is the most important way we fight data fires. No amount of performance tuning, index optimization, or wizard running will bring back lost data. If backups are manual, or automated and never tested, or restores are only manual, you have a fire waiting to happen. Head Geek Tom LaRock sqlrockstar says that "recovery is Job #1 for a DBA". It certainly is important. A great DBA automates all backups and recovery. If you are recovering manually, you are doing it wrong.
Other places where you want automation is in monitoring and alerting. You want to know something is going on even before someone smells smoke, not when users are telling you the database is missing. If your hard drive is running out of space, it is generally much faster to provision more resources or free up space than it is to recover a completely down system. Eventually you'll want to get to the point where many of these issues are taken care of automatically. In fact, that's why they invented cloud computing.
Get Going, the Alarm Bell is Ringing!
Become the Best DBA: A Lazy DBA. Lazy DBAs automate the stuff out of everything. And lazy DBAs know that automating keeps databases from burning. They automate away the dumb mistakes that happen when the system is down, they automate test restores, they automate away the pain of not knowing they missed setting a parameter when they hit ENTER. They know how to get to the fire, they know what to do and they fix it.
The Best DBAs know when database are getting into trouble, long before they start burning down.
The Best DBAs don't panic. They have a plan, they have tools, they have scripts. When smoke starts coming out of the database, they are there. Ready to fight that fire. They are ready because they've written stuff down. They've trained. They've practiced. How many clicks would it take you to restore 10 databases? Would you need to hit up Boogle first to find out how to do a point-in-time restore? Do you know the syntax, the order in which systems have to be restored? Who are the other people you have to work with to fix this fire?
As a new DBA, you should be working on automation every day, until all that work frees up so much of your time you can work on performance tuning, proper database design, and keeping your database fire truck shiny.