In my last blog post, we talked about protecting data at rest and data in motion. Thanks for all the really good comments and feedback you left. I think they gave us some good food for thought, especially a few items I hadn’t talked about, including mobile device security and management. In this post, I want to take things in a slightly different direction and talk about health care policy and how it affects data availability.
After working on the insurance side of health care for a good part of a decade, it became very clear to me that business policy influence had created a mentality of, "Everything must be up 100% of the time," and in many ways, it was true. While supporting a nursing triage hotline, people often called in with potentially life and death situations. Obviously, the availability of the telephone system, which was network-based, was critical to operating our environment. Our contact center, also network-based, and the backend systems our triage nurses needed to access, were also critical. We couldn’t have an outage that prevented our callers from reaching the nurses they needed to speak to. Lives were literally in the balance.
So how does one go about ensuring that data availability is achieved and that services are operational to an extended period of uptime, beyond your typical business? The answer to that, my friends, is architecture. You can only achieve the levels of high availability that are required in a healthcare environment when you specifically design for it. And these kinds of designs usually come with a mighty big price tag. But before I get into that part of the conversation, let’s break this challenge down into three steps. How do we go about achieving this unprecedented level of uptime?
You design it to be redundant.
First, you gain a full understanding of your business requirements, which are most often non-technical in nature. Then you design a model, whether it be network or software application architecture, which removes any and all single points of failure. This ideally results in an architecture design that can lose one or more critical components while operations continue. Ideally, you do this without the end-user noticing. This might mean network infrastructure, telecommunications circuits, application servers... it really can be anything. If a component can fail, you need to understand the failure modes, and plan for how to mitigate them through a redundant design.
You design it to be maintainable, and you take a proactive approach to maintenance.
No environment can operate forever without maintenance. You need to have a strategy in place for dealing with failed components or applications, and one that also allows you to take proactive measures to prevent future service disruption. This can mean an end of lifecycle hardware replacement, application software patching, or any other standard maintenance task. Maintenance should be routine and have time allocated for it. Simply saying, "You can’t have a maintenance window" isn’t going to fly. So, forget that illusion right now.
You figure out how to monitor it so you can react before service impact occurs.
The final key to preparing an environment to be highly available is to monitor it. You must first know what "normal" looks like to determine what "abnormal" is. This applies equally to network performance as it does software application performance. This is always a moving target, and it’s a lot of work. There are a lot of really good off-the-shelf software packages that can help with the basics (insert shameless, unsolicited plug for some of the cool SolarWinds tools here), or you can develop your own monitoring solutions. I’m not going to tell you what to monitor or how to do it, but I’m going to tell you that you need to figure out the answers to those questions and take the action appropriate for your environment.
Wrapping this discussion up, I know that achieving a truly highly-available IT environment sounds kind of like the Holy Grail, right? In many ways, it can be. I don’t know that you’ll ever achieve 100% of every one of these goals, but this is what you strive for, and how you need to approach it.
What do you think about the availability of IT services within your healthcare organization? What have some of your key challenges been? How are you addressing them? Do you have any tips, tricks, or battle scars to share that can help the rest of us? I'd love to hear your thoughts!
Until next time….