Alerts / Events from SNMP Traps

Please help! I've been scouring Thwack posts for days trying to figure out the best way to accomplish what feels like it should be a simple task, but I just can't wrap my head around it. Hopefully some SW pros can take a quick look and provide thoughts on the best approach, and possibly a bit of help.

Here's our scenario: Working with a custom MIB that has been uploaded to the Solarwinds MIB database. There is a single device (source IP) that sends SNMP traps for the entire system.

In all cases, there is a trigger OID, and a clear OID. Within each trap message are 10 varbinds the contain the critical details necessary to troubleshoot and resolve the issue such as 'location', 'node', 'interface', 'buscode', 'port number', etc.

There are 2 generic notifications (one trigger, one clear) where the only way to determine the issue is to examine one of the varbinds.

generic trigger:

generic clear:

In addition the MIB has described 15 specific issues and has a trigger/clear OID for each one. For example:

OID for 'TCP Connection Failed' trigger

OID for 'TCP Connection Cleared' clear

I need to figure out how to:

1) create an event in the event log from a trap

2) create an advanced alert from a trap

Any help would be really appreciated.

Find more posts tagged with

traps

help me

npm

trap alerts

Accepted answers

All comments

RichardLetts

I answered an almost identical question to this a couple of months ago:

see: Custom SQL Alerts based on the SNMP trap message content

have a look at that thread, and ask questions.

tkercher

Richard, thanks so much for your response. I have been working from the post you're referring too, and it fits the scenario where I have a generic trap and the trigger/clear value comes from a varbind. My problem is the other scenario, where I have one source (IP address), the MIB provides an OID for trigger, and a different OID for clear. So for example, I could receive multiple traps triggers for TCP failure, and in the trap message it would indicate different nodes, interfaces, ports. The clears are a different trap, again from the same source, that also indicate the specifics in the varbind. I've played with the idea of tagging the trigger traps with a combination of varbinds such as node+interface+portnumber, and then writing a clear that looks for the clear trap and the same combination of varbinds, but I'm afraid I'm just not that skilled with SQL. I'm not even sure this is the best, most efficient approach, but trying to learn SQL and get this done at the same time and need help.

Thanks,
Tom

RichardLetts

step 1: colour the trigger and reset traps differently so the start to look like this:

OID for 'TCP Connection Cleared' clear:

[you pasted images instead of text, so I just used color fill from paint making these look ugly, but hopefully you get the idea.]

putting the combination of things to match in the 'tag' sounds like the right approach.

can you paste the output [as text] of

SELECT TOP 10      [DateTime]      ,[IPAddress]      ,[tag]      ,[Hostname]      ,[NodeID]      ,[TrapType]      ,[ColorCode]  FROM [SolarWindsOrion].[dbo].[Traps]  where Tag <> '' and TrapType like 'SIRIUSXMSTREAMING%'  order by datetime desc

Once you have the trigger and clear trap correctly colored share your custom SQL [AS TEXT] and I can look at it.

tkercher

Richard,

I ran the attached query and the results are as follows:

8/15/2016 4:45 PM	10.15.140.16	SWTMC1	10.15.140.16	1600	SIRIUSXM-STREAMING-MIB:buscodeDataAbsentClear	3992576
8/15/2016 4:45 PM	10.15.140.15	SWTMC1	10.15.140.15	1595	SIRIUSXM-STREAMING-MIB:buscodeDataAbsentClear	3992576
8/15/2016 4:44 PM	10.15.10.19	XMCOUN	10.15.10.19	1596	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.19	SWCOUN	10.15.10.19	1596	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.19	SRCOUN	10.15.10.19	1596	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.20	SWCOUN	10.15.10.20	1599	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.20	SRCOUN	10.15.10.20	1599	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.20	XMCOUN	10.15.10.20	1599	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.20	XMCTGB	10.15.10.20	1599	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232
8/15/2016 4:44 PM	10.15.10.20	SRCTGB	10.15.10.20	1599	SIRIUSXM-STREAMING-MIB:mdsNotif.51	232

tkercher

Trigger condition:

where nodeid in

(SELECT a.nodeid

FROM traps a with (nolock)

where colorcode=232 and acknowledged=0

and not exists ( select 1 from traps b where a.nodeid=b.nodeid and a.tag=b.tag and b.colorcode=38144 and b.datetime>a.datetime)

)

Reset condition:

where nodeid not in

(SELECT a.nodeid

FROM traps a with (nolock)

where colorcode=232 and acknowledged=0

and not exists ( select 1 from traps b where a.nodeid=b.nodeid and a.tag=b.tag and b.colorcode=38144 and b.datetime>a.datetime)

)

tkercher

Right now I have these traps tagged with the 'buscode', but ideally I'd like to tag each trap with a combination of several varbinds (node, interface, port, buscode). Then, adjust the queries to match

RichardLetts

that looks good, Note, I don't understand the traps in this case... I'm going to have to guess in the stuff below

If you want to have separate alerts for each BUSCODE (is that the type of error?) then you might have to join with the trapvarBinds table.

If you use the queries you have below then you get ONE alert type per NODEID -> this might not be what you want if you want to have one alert per MDS NODE (assume this is the thing inside the system being monitored)

If you want the latter then you will have to create nodes that match the MDS Node values [I do not know if the external node type will work for this] and then match the node caption to the tag data