Skip navigation

Geek Speak

4 Posts authored by: airbornegeek

A couple of weeks ago, I wrote about how I love documentation and how useful it can be in the future, for you or someone else. I also lamented slightly about how it is hard on most projects to get the time to write good reference documentation. This week, I'm going to discuss one possible way to knock out some of the more time-consuming parts.

 

The key word is: Design. That is--the "Design" phase of the project. In most projects, regardless of the flavor of SDLC utilized, there is a block of time to be utilized for designing the solution. Since this is work that will be happening anyway, getting some good documentation out of it is simply a matter of writing down what gets discussed, decisions that get made (along with the "why"s, of course), and how the solution(s) will be structured. Chances are these things get written down, anyway, but outside the mindset of their possible use as future reference material. Usually, by its nature, designing a project will produce some useful artifacts; things like high-level architecture diagrams or possibly an ERD or two. If it's a data-integration or BI project, and one is into details, source-to-target mappings are likely generated at this point.

 

All of these items add up to a decent set of notes for the future, explaining the solution and where it came from. This way, even if no dedicated time can be worked into a project for documentation, useful items can be produced.

 

I think there's another benefit to this course of action. I have a phrase I use to describe my philosophy on this topic: I don't like to think while I'm writing ETL. This sounds kind of bad on the surface, but what I really mean is this: When it comes time to actually sling some code, I want to be able to concentrate on the mechanical details; Things like doing everything right, following best practices, security concerns, and making things as highly-performing as they can be, all right from the get-go. With the data flows already drawn out and the business rules documented and agreed upon, it is easy to focus on those things.

 

We had a fairly nice conversation about it being hard to get the time to write documentation when I wrote about it before. Would beefing up the efforts during the design phase help? Are ya'll even able to get good time for the design stage in the first place (I've been in predicaments where that gets axed, too)?

Data people and developers don't really get along very well. There have been lots of reasons for this historically, and, I'm guessing, as time goes on, there will continue to be conflict between these two groups of people. Sometimes these disagreements are petty; others are more fundamental. One of the areas I've seen cause the most strife is shared project work using an Agile software development lifecycle. I know talking about Agile methodologies and data-related projects/items in the same sentence is a recipe for a serious religious battle, but here I want to keep the conversation to a specific couple of items.

 

The first of these two items is what happens when an application is developed using an ORM and a language that allow the dev team to not focus on the database or its design. Instead, the engineer(s) only need to write code and allow the ORM to design and build the database schema underneath. (Although this has been around for longer than Agile processes have been, I've seen a lot more of it on Agile projects.) This can lead to disconnects for a Development DBA-type person tasked with ensuring good database performance for the new application or for a Business Intelligence developer extracting data to supply to a Data Mart and/or Warehouse.

 

Kind of by its nature, this use of an ORM means the data model might not be done until a lot of the application itself is developed…and this might be pretty late in the game. In ETL Land, development can't really start until actual columns and tables exist. Furthermore, if anything changes later on, it can be a lot of effort to make changes in ETL processes. For a DBA that is interested in performance-tuning new data objects/elements, there may not be much here to do--the model is defined in code and there isn't an abstraction layer that can "protect" the application from changes the DBA may want to make to improve performance.

 

The other problem affects Business Intelligence projects a little more specifically. In my experience, it's easy for answers to "why" questions that have already been asked to get lost in the documentation of User Stories and their associated Acceptance Criteria. Addressing "why" data elements are defined the way they are is super-important to designing useful BI solutions. Of course, the BI developer is going to want/need to talk to the SMEs directly, but there isn't always time for this allotted during an Agile project's schedule.

 

I've found the best way to handle all this is focusing on an old problem in IT and one of the fundamental tenants of the Agile method: Communication. I'll also follow that up with a close second-place: Teamwork. Of course, these things should be going on from Day 1 with any project…but they are especially important if either item discussed above are trying to cause major problems on a project. As data people, we should work with the development team (and the Business Analysts, if applicable) from the get-go, participating in early business-y discussions so we can get all of the backstory. We can help the dev team with data design to an extent, too. From a pure DBA perspective, there's still an opportunity to work on indexing strategies in this scenario, but it takes good communication.

 

Nosing into this process will take some convincing if a shop's process is already pretty stable. It may even involve "volunteering" some time for the first couple projects, but I'm pretty confident that everyone will quickly see the benefits, both in quality of project outcome and the amount of time anyone is "waiting" on the data team.

 

I've had mixed feelings (and results) working this type of project, but with good, open communication, things can go alright. For readers who have been on these projects, how have they gone? Are data folks included directly as part of the Agile team? Has that helped make things run more smoothly?

I have a confession: I like documentation.

 

I know.

 

But I have good reasons. Well; some reasons are better than others. I like to type (not really a good reason); I like to write (although I'd argue I'm not all that great at it); and I like to have good references for projects I've done that will hopefully be useful to any future person--or myself--who has to support what I've put together (OK, this is actually a good one).

 

When talking about data projects or DBA-related work, reference documentation can contain a lot of nitty-gritty information. There are two specific areas that I think are most important, however. The first of these is useful in the case of Business Intelligence projects, and usually takes the form of a combination Data Dictionary/Source-to-Target Mapping listing. The other is in the vein of my post from last week wherein I discussed asking the question of "why" during the early stages of a project. Even having only these "why" question and answers written down can and will go a long way towards painting the picture of a project for the future person needing to support our masterpieces.

 

As good as it truly is for all people involved in a project, writing reference documentation is something that isn't necessarily easy or a lot of fun to do. One of the most frustrating things about it is that in a lot of instances, time to write this documentation isn't included in the cost or schedule of the project; especially, in my experience, if it's following something like the Scrum methodology. In the case of a "day job", spending time on documentation may not have the same immediate need, as the people who participated on the project theoretically aren't going anywhere. Furthermore, people spending that time takes time away from other work they could be doing. In the case of an outside consultancy, paying an outside firm to sit around and write documentation has an obvious effect on the project's bottom line.

 

If I had to guess, I'd say most of you readers out there aren't as into documentation as much as I am…I really do understand. But at the same time, what do you do for the future? Do you rely on comments in code or something like Extended Properties on database objects?

And, for those of you that do attempt to provide documentation… do you have to make special efforts to allow the time to create it? Any secrets to winning that argument? ;-)

Over the last few months, I've spent a lot of time talking to different people about the importance of the "B" in BI--as in "Business Intelligence" (stay with me, DBAs, I'll get you roped in here soon enough). I joke that that "B" is there for a reason and it's no accident that it's the first letter in the acronym. The point I make is that the hard part of BI projects isn't deciding what technology to use or working through hurdles that come from questionable data, but instead, understanding the Business needs that got the project started in the first place.

 

Or, put another way, making sure that the "Why?" questions are asked first and foremost, and that the answers to those questions are used to ask better "What?" questions on down the line while the solution is designed, developed, and implemented.

 

This works well when the word "Business" is directly included in the type of data work I do, but this applies just as readily to plain ol' DBA work, as well. Everything from data modeling considerations to planning big database consolidation projects need to start out by asking and understanding the "Why"s coming from our business leaders and our userbase. Sometimes these questions are easy--"Why do you want to be able to restore yesterday's database backup?" Other times they might be hard to get answers to--"Why is this KPI calculated three different ways in this one report?" (true story!). Chances are, asking these questions will have some kind of monetary impact. For a BI project, it may cause the project to complete quicker with less re-work late in the game, while the need to upgrade some old, unreliable hardware may lead to a server virtualization project for a DBA team.

 

I've seen both BI and DBA projects go sideways because the right questions weren't asked early enough in the project lifecycle; conversely, I've worked with some great BAs who are fantastic at understanding the business rationale and ensuring we, the technology team, all did, too.

 

What kind of save-the-day stories do you have that started out by asking "Why?"

Filter Blog

By date: By tag: