Email alerts not coming through

So recently we had alerts stop for two days which we have had happen before. In the past we have resolved the issue by restarting the Solarwinds server, however we would like to find out what is causing the issue. Can someone point me in the right direction as where to look/check, maybe log files or somewhere similar? I checked the services and they were all started and running and I couldn't seem to find anywhere else to check. Some basic steps, procedures would be a great help for when this happens again, so we can finally find the problem.

I've read some previous things like the wrong time being set and alerts being disabled; things like that (which haven't been the issues) but don't really know where to look now to be honest.

Thanks

Find more posts tagged with

Alerts

Accepted answers

All comments

silverbacksays

Hi Matt,

When the emails were not getting through, were you seeing events/alerts coming through your SolarWinds instance?

As a starter for 10:

I've had something similar happen in the past. If the primary poller runs out of disk space, it'll stop processing alerts, which will in turn stop any advanced alerts firing, hence stopping the emails which are sent out as one of your trigger conditions of the advanced alerts you use. That's the first thing I would look at.

matt_rees

So I just checked the alerts through Solarwinds like you said and I am now confused.

There are alerts that did seem to trigger over the two days whilst the emails stopped. However, there are a lot of alerts that seem to be missing, even now (even though it's up and running). I went back to the last email that did come through fine and 5 or 6 emails before that, and Solarwinds shows no records of those alerts either any ideas what that means? It appears to be just showing random alerts, throughout the day, not every alert.

Also, i've checked the server and I have around 2GB data left on the C: drive. Just to let you know lastly, I managed to get the alerts to come back up this time by stopping the Solarwinds services and restarting them, but I couldn't tell you which one brought it back up,

Thanks

silverbacksays

What version of NPM are you running? One thing which comes to mind is that you may be having issues getting SNMP responses back from your target nodes, and hence no data to generate alerts/events from, but you're not aware of it due to NPM calculating availability from ICMP responses.

You can run a report to find out if this is happening. Go to your report writer (I'm using the classic report writer in this example, that you execute from your primary poller, as it's available to both older and newer versions), create a new 'Current Status of Nodes, Volumes, etc.' report, and add in node id, node name and last sync (all from the Network Nodes section). Change the filter to show everything where last sync was more than say, one day ago, and where the SNMP Community field is not empty (so you're only seeing SNMP nodes) and preview.

Everything you see there is not responding to SNMP.

EDIT: If you have a large number of nodes, you may want to add in 'Status is not equal to Unmanaged' to your filters, to save you panicking when the report pulls back a load of unmanaged nodes

matt_rees

We are currently on version 10.7.

Well we only use SNMP on routers to be honest, the rest of our nodes are monitored through ICMP. When checking the alerts it shows both ICMP and SNMP results, but as I said they are random. For example a few days ago it only shows around 8 alerts on the alert page, but when I check the mailbox we had way more than that. I've made sure that 'all alerts' and 'all network objects' and 'all device types' are displayed but it's still the same.

I don't think this is linked to the emails completely stopping but it is definitely highlighting another issue.