Turn off the alerting, turn off the monitoring. Information is power.

I've recently been in the situation where I have introduced monitoring onto SQL databases and onto an environment including a SAN. The customer has told me firstly that the alerts being generated are too noisy, then secondly that we should stop access to individuals and stop monitoring all together on some databases.

Could I reduce the number of alerts?

I have gone back to the customer and pointed out that the alerts are valid and I have checked through the thresholds with the customer and we have confirmed that they are valid thresholds.

They were receiving emails every day to tell them about problems with databases and nodes.

Database access denied.

The SQL server databases that I was monitoring suddenly stopped being monitored and then the DBA's came and told me that the Solarwinds account no longer had access due to excessive SQL reads on the cluster and this was causing a problem for the databases.

One of the project managers then came to me, to ask me why his report on SQL databases had stopped working?

Embarrassed technical engineers.

The SAN was recently monitored too, with SRM, it highlighted numerous issues as soon as it was put in, all the time during meetings the technical engineers were assuring the managers and directors that the SAN was fit for purpose and there was plenty of space.

One of the managers who had Solarwinds access logged in to Solarwinds and saw the state of the SAN and the problems it had connecting to a SQL database that had particular latency issues. After the meeting I was told to stop access to SRM for this particular manager.

Solarwinds is great and information I gather about our infrastructure is invaluable. Sometimes Solarwinds can be its own worst enemy.

I am discovering that I need to be a politician as well as a monitoring guy to keep the monitoring alive and to give people the information they need. Different people within the organisation have different requirements and I have to juggle their needs.

Has anyone else experienced this?

Find more posts tagged with

dba

information

sql

solarwinds

srm

Alerts

project

Accepted answers

All comments

mldechjr

There are several good video servies on tuning Alerts with time delays, resets and nested alert conditions. I am sure that one or more of these features would be useful to you. Check out this series:

https://support.solarwinds.com/success_center/network_performance_monitor_(NPM)/network_performance_monitor_training

The Far left Video set is free to watch - Section 11 on Alerts sounds like your target. There are also some free Webinars in Thwack that get into even more granular explanation. Sorry not to answer specifics but you gave none. Getting into advanced alerts is where you need to go.

FormerMember

Yes they all want the alerts to see whats happening, but then they don't when the alerts get a bit too much for them to handle. Its a difficult balancing act. Thanks for the info. I will run through the vids.

meech

Hi acmtix,

Check out anything Head Geek adatole writes on monitoring--he even has a free book on it. And let us know how it goes!

Jfrazier

A balancing act it is...

The art is being able to help temper the volume/quality of alerts in a manageable format that makes sense to the requestor.

Loop the recipient in as to what you are doing and show how you can help them do their job easier and more efficiently. Make

sure they know what sort of alerts they may be getting and have agreed to getting them. If alerts are to be sent to another party

to provide action upon them, do not create/implement them until the other party is in board with it and has the proper documentation.

The more power you use to focus on a particular thing to monitor for problems can cause harmful impact to the environment.

A simple analogy. To look at something smaller you need a more powerful microscope. The more power or magnification you use may require more light. The additional light may harm or impact what you are trying to look at.

I approach all this by having meetings with the group whose stuff I am about to monitor. I start off with their pain points...what do they need right now. Then working with them to catch the other stuff later on. This way we are not introducing noise....

gfsutherland

Great points... well said!

FormerMember

Yes, i've had requirements meetings with most of the departments, mapping out their expectations.

I am replacing an existing monitoring system with Solarwinds and they were entrenched in this system so that has been a challenge to move them away from the old system.

The company is a large worldwide org that manages other companies information and the requirements have grown and grown and grown for the Solarwinds monitoring environment.

They don't have an asset database and trying to work out what they have got in terms of environments has been a challenge. I keep finding new vcenters and new subnets filled with servers all over the world.

They have distribution lists setup for Solarwinds, but I looked at one yesterday and there was a guy in there who left the company 2 years ago...

Its fun, there is a lot to do and its a challenge trying to match their requirements to what Solarwinds can do for them.

You mentioned pain points, they have so many here its difficult to keep up... but I will win!

Jfrazier

I can relate as I have lived that life several times so I feel your pain.

tinmann0715

Many times, and it is indeed frustrating and demoralizing. Monitoring can fall easy prey to be labeled the scapegoat for the real issue at hand. And while the passionate, demonstrative, inspired defense might work on the big screen it can often only compound the issues. In my many years of monitoring I have learned to swallow the criticism and come forth with, "How can we do better to solve our problem together? What more can monitoring do to make you happy, Mr./Ms. Customer?" And then sit back and listen. Resist the urge defend. You can counter a point, but do it dispassionately. No victim statements either. No one likes that.