In a previous post I discussed the importance of having an intelligent communication system automate the process of sending alerts and managing escalation in the event of a large-scale IT emergency.
Let’s assume a regional emergency takes down power for an indefinite period. You have your production systems replicated in two geographically distinct datacenters. Though your primary site is in the impacted region, the datacenter’s generators give you some time to perform a graceful switchover to the other secondary site.
Telecommunications in the impacted region may be sporadically or unevenly available. For this reason—among others—you chose as your intelligent notification system a third party service (SaaS) whose systems are appropriately redundant in regions different than yours.
As your NOC kicks-offs alerts to operations teams, your notification system looks up relevant on-call contacts in all the team calendars and sends out initial, secondary, and tertiary calls, pages, emails, text messages, as needed, automatically steering contacts into appropriate chat sessions and conference calls, and automatically escalating alerts up the chain within each team. Concurrently, higher level operations managers are similarly alerted and join forums to make a go/no-go decision to switchover every 15 minutes, based on estimates of lost data—currently in the replication pipeline—and lost revenue due to unavailability of services should the primary site fully go down.
More Channels are Better
Based on device and carrier types, members of your different NOC and operations teams may have different access to communications channels. As a result, you should leverage as many channels as possible.
In this sense, though it cannot substitute for a full-scale intelligent communication system, Twitter effectively could help you bridge gaps. Twitter offers a highly scalable system that supports any device with the software to send and receive tweets.
In this case, you need to restrict communications, which runs at cross purposes with Twitter’s vast majority of tweeters, who broadcast their short messages to whoever will listen. You can setup a Twitter group with privacy that only the NOC and operations teams follow and simply disapprove any other unwanted followers. Posting all messages to the group in direct mode keeps all communications private.
The 140 character limit for each tweet will of course hinder the continuity of communication over this channel. But tweets would be perfect for short progress updates on specific tasks.