Check your hardware and make sure they're not running hot also check your polling and stats collection intervals and the number of elements your monitoring as these can all cause "white-outs" in data either individually or a combination.
We were having a similar problem after we added two new switches and what we found was we were polling just under 9000 elements and polling every 2 minutes and collecting stats every 10 minutes and the hardware couldn't cope. So we ended up moving some of the elements to another NPM and also changing the polling and stats collection timers and everything started working again.
We've found out that one of the disks in our SQL Server was running out of disk space. Also, we found out that one of the devices that we're monitoring was sending loooooooaads of syslogs to our NPM (we received 480 syslog-messages from one device each second!).
Obviously, the large number of syslogs caused the DB to fill-upems like and eventually this has led to the disk running out of disk space. We freed some disk space up, and now everything seems to work OK.
Now we're facing the following challenges:
- how to clean up the DB from those unneeded syslogs?, and
- how to detect problems like these earlier than a disk running out-of-space?
For now, please be informed that it seems like we have found the source of our probleem. And you're right, it had to do with hardware...
- Change the syslog retention to 7 days
- Run database maintenance
- Set up alerts on critical disks
Thanks for these suggestions!
Have been trying to find the "Syslog retention" parameter, but cannot seem to find it. All I see, is "Event retention". Where do I have to look?
Syslog Server Settings
Use the following procedure as a guide to starting and configuring the Syslog
To start and configure the Syslog Viewer:
1. Click Start > All Programs > SolarWinds Orion > Syslog and SNMP
Traps > Syslog Viewer.
2. Click File > Settings.
3. Click the General tab in the Syslog Server Settings window.
4. Adjust the Maximum number of messages to display in Current
Messages view slider to set the number of messages you want to display.
5. If you want to Automatically Refresh the Current Messages View, check
the option accordingly, and then set the refresh rate with the middle slider.
6. Adjust Retain Syslog messages for how many days? to set the length of
time Syslog messages should stay in the database.
Ah, great! I was only looking in the web interface of NPM.
Have changed the retention to just one day now, in order to clean up the database overnight.
Thanks for support!
Can you tell me about the Orion environment?
How many elements?
Where is the SQL server and what is it's set up?
Do you have other modules running?