5 Replies Latest reply on May 8, 2012 10:54 AM by cmgurley

    Monitoring Solution or Ecosystem?

    Mrs. Y.

      Recently I've noticed an interesting phenomenon as I start to talk to other IT professionals about their logging and event monitoring systems.

      I like to call it the Shadow Effect. It's the gap between the products that management says are being used by the organization and the reality

      of what the engineers actually deploy to get their jobs done.

       

      There are the "official" products that an organization purchased to magically solve the problem of alerting and logging. They're usually

      very expensive, took hundreds of hours to implement and are generally reviled by the rest of the IT staff. The vendor convinced senior

      management that the product was supposed to work “out of the box” and the professional services contract was only an add-on in order to get it

      up faster and for knowledge transfer. Eventually it quietly dies when the behemoth support contract comes up for renewal and management finally sees that it's

      gathering dust along with the old PS2 keyboards. This is mostly because the vendor’s consultants didn't configure the application properly or

      you need a degree in theoretical physics to use and maintain it. The CAPEX is simply written off as a failed experiment and there's a quiet

      wake for yet another product gone to the enterprise software graveyard.

       

      Then there are the REAL solutions the staff uses to get their jobs done. These products don't really work very well, throwing lots of false

      alerts, sometimes missing the critical stuff, and bringing about a vague yearning for something better. They might be open source or custom

      in-house solutions, installed by one guy who had a solid understanding of it, but who doesn't even work there anymore and hasn't for a while.

      So everyone tiptoes around the system, saying, "Don't touch it," afraid of the alternative: no solution. The engineers usually fight the hardest

      at finding a replacement too, because they have Stockholm Syndrome. They've identified with their captors, fully convinced that getting 228

      false alarms via email every night isn't all that bad.

       

      But why can't there be something better? Since when does a monitoring solution have to become it's own ecosystem?

        • Re: Monitoring Solution or Ecosystem?
          nicole pauls

          More food for thought: I saw a presentation at the RSA Conference that was discussing the use of Enterprise SIEM/Log Management products and how to roll them out. One of the presenters had developed this complex prioritization system for the massive amount of notifications generated on a regular basis from one of said enterprise systems. There's something to be said for understanding how to prioritize information, but if your answer to generating hundreds of notifications/incidents per day is better prioritization and not "fix the thing generating them" it sounds like you need a better mousetrap.

          • Re: Monitoring Solution or Ecosystem?
            byrona

            Mrs. Y

             

            After reading this it instantly made me think of where our company used to be (the Stockholm Syndrome is such a true thing that I couldn't help but laugh).  Some of these problems still exist now but it's much better since we rebuilt our monitoring infrastructure on Orion and got buy-in from all of the technical teams as well as the executive team.  I really think that the buy-in is key and the executive team needs to enforce it by making it clear that all technical teams will be consolidated on the same set of tools.  We continue to try and weed out the false positives (noise) from the system but I think that is a never ending fight when you are in a constantly changing environment such as ours.

             

            However, I think a successful monitoring system does have to become an ecosystem.  A successful monitoring system should be touching every aspect of your company pulling all of your technical people together as well as your executive team.  It should monitor and collect data from all of your systems, provide a single pane of glass and centralized place for all of your technical people to look for both making decisions and solving problems and it should provide reports to help the executive team making important decisions regarding the company; doesn't that quality as an ecosystem?

              • Re: Monitoring Solution or Ecosystem?
                Mrs. Y.

                It's good to know that you've had some success with your monitoring infrastructure. But I believe that IT has to be run more like a benevolent dictatorship than a democracy. Too much consensus can destroy these types of initiatives.

                 

                Ideally, I think a good monitoring system should integrate seamlessly with the rest of the infrastructure or the IT ecosystem, not become it's own. Whenever the latter occurs, it becomes a thing that seeks to justify and validate itself, not providing ROI for the organization.

              • Re: Monitoring Solution or Ecosystem?
                cmgurley

                Mrs. Y.,

                 

                In this thread, you wrapped up two terms that I've found elusive to combine effectively--"logging" and "monitoring". The latter of the two seems to have a proud showing from multiple vendors (we use two here; SCOM & Orion) and forecast is looking pretty sunny, IMO. Some are agent based, others poll, and all in all they show up/down/% info pretty well. Think of that as medical vital signs.

                 

                Now logging on the other hand has more of an emotional feel/value and has been about as hard to narrow down and prove effective as psychiatry is to objectively quantify. Vendors have advertised SIEMs for years that supposedly correlate and turn gobs of data into specks of information, but I've yet to see it fine tune (not necessarily to their discredit). The fact is, logging varies vastly by environment and is affected by a myriad of factors (logging levels on devices, mixture of devices, policies on network, etc).

                 

                In my environment, I'd love to see logging/SIEM as a value-add rather than an anchor-tied (and sinking), but it's a hard fight (with myself, much less others). We do pretty well in our admitted ignorance (from logging; we still have plenty of monitoring), and the time required to tune-watch-tune-repeat an SIEM (even LEM) is pretty steep.

                 

                For us, it may be a gradual approach. As I eval LEM right now, I'm thinking of putting just a foot in the water for a few network devices (firewalls and Linux-based devices) to see if we can make headway there. The thought of unleashing 7 million syslog messages (that currently pour into our Kiwi Syslog server from all of our systems) upon LEM and then trying to figure out correlation and rules while the alerts fly by is enough to drive one insane...

                 

                Monitoring? Yeah, that's an ecosystem that is ready to be broadened beyond IT. Logging, though? I'm not sure it's even a system yet, much less something life and order. The search is on, even so...

                 

                ~Chris