In my time at Tech Field Day, I've heard a lot of discussion about monitoring products. Sometimes these talks get contentious, with folks pointing out that a certain feature is "useless" while another (usually missing) one is absolutely critical for a product to be taken seriously. Then one day it clicked: Monitoring isn't one thing; there are lots of different tasks that can be called "monitoring"!
Let's consider storage. Administrators are concerned with configuration and capacity. IT management is worried about service levels and cost. Operations worries about failures and errors. The vendor has a whole set of parameters they're tracking. Yet all of these things could be considered "monitoring"!
Realizing that monitoring means different things to different people, I've come to look at monitoring tools differently too. And it's really brought them into focus for me. Some tools are clearly designed for IT, with troubleshooting and capacity planning as the focus. Others are obviously management-focused, with lots of pretty charts about service level objectives and chargeback. And so on.
Given the diversity of these tools, it's no wonder that they appear controversial when viewed by people with different perspectives. Systems administrators have a long-standing disdain for cost accounting tools, so it's no wonder they flinch when presented by a tool that focuses on that area. And they'd be sorely disappointed if such a monitoring package didn't show hardware errors, even though these wouldn't be appropriate for an IT management audience.
So What Cha Want?
Without getting too "zen" I think I can safely say that one must look inside oneself before evaluating monitoring products. What features do you really need? What will help you get your job done and look like a superstar? What insight are you lacking? You must know these points first before you even consider looking at a flashy new product.
And some products sure are flashy! I'll admit, I've often been sucked in by a sleek, responsive HTML5 interface with lots of pretty graphics and fonts. But sometimes a simple text-mode tool can be much more useful! Never underestimate the power of "du -sk * | sort -n", "iostat -x 60", and "df` -h"! But high-level tools can be incredibly valuable, too. I'd rather have a tool surface the critical errors than try to "awk" my way through a log file...
Consider too whether you're ready to take advantage of the features offered. No amount of SLO automation will help you develop the SLA's in the first place. And does any company really have an effective cost model? The best tools will help you build understanding even if you don't have any starting inputs.
I can't tell you how often I've tried out a new monitoring tool only to never return to look at the output. Monitoring isn't one thing, and the tools you use should reflect what you need to accomplish. Consider your objectives and look for tools that advance them rather than trying out every cool tool on the market.