Recently I've noticed an interesting phenomenon as I start to talk to other IT professionals about their logging and event monitoring systems.
I like to call it the Shadow Effect. It's the gap between the products that management says are being used by the organization and the reality
of what the engineers actually deploy to get their jobs done.
There are the "official" products that an organization purchased to magically solve the problem of alerting and logging. They're usually
very expensive, took hundreds of hours to implement and are generally reviled by the rest of the IT staff. The vendor convinced senior
management that the product was supposed to work “out of the box” and the professional services contract was only an add-on in order to get it
up faster and for knowledge transfer. Eventually it quietly dies when the behemoth support contract comes up for renewal and management finally sees that it's
gathering dust along with the old PS2 keyboards. This is mostly because the vendor’s consultants didn't configure the application properly or
you need a degree in theoretical physics to use and maintain it. The CAPEX is simply written off as a failed experiment and there's a quiet
wake for yet another product gone to the enterprise software graveyard.
Then there are the REAL solutions the staff uses to get their jobs done. These products don't really work very well, throwing lots of false
alerts, sometimes missing the critical stuff, and bringing about a vague yearning for something better. They might be open source or custom
in-house solutions, installed by one guy who had a solid understanding of it, but who doesn't even work there anymore and hasn't for a while.
So everyone tiptoes around the system, saying, "Don't touch it," afraid of the alternative: no solution. The engineers usually fight the hardest
at finding a replacement too, because they have Stockholm Syndrome. They've identified with their captors, fully convinced that getting 228
false alarms via email every night isn't all that bad.
But why can't there be something better? Since when does a monitoring solution have to become it's own ecosystem?