Syslog alert regex not working

TL;DR - regex parsing in the old tool seems broken, but it seems work-aroundable by using SWQL queries in the web-based alerting UI. Thanks jm_sysadmin!

I'm trying to set up a syslog alert with a regex that uses a negative lookahead that should hit on anything containing string1 and not (string2 or string3). This works in the testers in regex101.com and regextester.com, but in the alert, it does not trigger at all.

I have a catchall alert that just looks for string1 without the other stuff, and that works fine, but it's noisey.

(?:(?!.*Gi3\/0\/27 and port Gi3\/0\/23)(?!.*Gi3\/0\/23 and port Gi3\/0\/27))flap

This should hit on anything that contains "flap" and does NOT contain "Gi3/0/23 and port Gi3/0/27" or "Gi3/0/27 and port Gi3/0/23"

E.g. if I do a "send log flap flap flap" on the device, this regex should hit. It doesn't.

And yeah, unfortunately, some flapping is expected, the way some of our wireless controllers currently work.

Example of how I want it to work:

...but it never hits on anything, as far as I can tell. I have tried adding "/" at the beginning and "/g" at the end of the definition - doesn't help.

Anyone know of a way to get this going?

Find more posts tagged with

syslog

not working

negative lookahead

regex

Accepted answers

jm_sysadmin

That's a good idea, since I have Trap data I tried:

SELECT TOP 100 NodeID, IPAddress, Caption, Nodes.Traps.DateTime, MINUTEDIFF(Nodes.Traps.DateTime, TOUTC(GETDATE()) ) as [Minutes]

FROM Orion.Nodes Nodes

Where Nodes.Traps.DATETIME is not Null and MINUTEDIFF(Nodes.Traps.DateTime, ToUTC(GETDATE())) < 5

Which works for traps in the last 5 minutes. So your Syslog last 5 minute filter should be:
Where Nodes.Syslogs.DATETIME is not Null and MINUTEDIFF(Nodes.Syslogs.DateTime, ToUTC(GETDATE())) < 5 And Nodes.Syslogs.Message like '%flap%' and Nodes.Syslogs.Message not like '%Gi3/0/23 and port Gi3/0/27%' and Nodes.Syslogs.Message not like '%Gi3/0/27 and port Gi3/0/23%'

You mentioned not being able to see where the field is, that is a great part about SWQL studio from the SDK. I'd explain it, but micheal100 did a great job in the post below.
Intro to SWQL Studio

But basically it lets you sift through all the SWQL data available.

Let me know if you have any questions or if this gets the job done.

All comments

jm_sysadmin

I don't currently have any Syslog sent into Orion so I may have made a mistake and can't test, but maybe the SWQL Alert can help? Choose SQWL alert and node, and use the where statement below and see if that helps.

Where Nodes.Syslogs.Message like '%flap%' and Nodes.Syslogs.Message not like '%Gi3/0/23 and port Gi3/0/27%' and Nodes.Syslogs.Message not like '%Gi3/0/27 and port Gi3/0/23%'

If you have SWQL Studio from the SDK you can test my query directly.

SELECT TOP 100 Nodes.NodeID, Nodes.Caption, Nodes.Syslogs.Message

FROM Orion.Nodes Nodes

Where Nodes.Syslogs.Message like '%flap%' and Nodes.Syslogs.Message not like '%Gi3/0/23 and port Gi3/0/27%' and Nodes.Syslogs.Message not like '%Gi3/0/27 and port Gi3/0/23%'

Hopefully that will help

neomatrix1217

You can accomplish this very easily with Orion Log Analyzer

Log Management and Analysis Software | SolarWinds

jaybone

Thanks - I didn't even realize that web-based alerting could see syslog stuff, actually. I've been still using the old Syslog Viewer executable based alerts.

As-written, your query seems to get extraneous stuff, though. Every time the trigger is evaluated, the alert fires, apparently because it's looking at all the syslog messages that have even been received. I'm thinking I'd have to make it look at the timestamp of the message, as well. E.g. make it only look at stuff that has come across in the last X minutes, matching the trigger evaluation interval.

Problem there is, I'm not sure how to find out what 'Nodes.Syslogs.whatever' actually maps to, in order to find the timestamp field. There is no table or view named that in the database. It appears to be tied to the dbo.syslog table somehow, but I have no idea if I'd use the 'DateTime' field from that table, nor how to do date/time math in SQL to tell it "only look at stuff timestamped within the last X minutes."

It also doesn't seem to want to let me change the select part of the statement, so I can't grab the message text with this. Any idea if there's a way to do so?

jm_sysadmin

That's a good idea, since I have Trap data I tried:

SELECT TOP 100 NodeID, IPAddress, Caption, Nodes.Traps.DateTime, MINUTEDIFF(Nodes.Traps.DateTime, TOUTC(GETDATE()) ) as [Minutes]

FROM Orion.Nodes Nodes

But basically it lets you sift through all the SWQL data available.

Let me know if you have any questions or if this gets the job done.

jm_sysadmin

One more reply about how to get the message text. Since we have it in the Syslog table we can get it.

SELECT TOP 1000 MessageID, EngineID, DateTime, IPAddress, Acknowledged, SysLogFacility, SysLogSeverity, Hostname, MessageType, Message, SysLogTag, FirstIPInMessage, SecIPInMessage, MacInMessage, TimeStamp, NodeID, ObservationSeverity

FROM Orion.SysLog

We aren't alerting by the Syslog entry, which might be good so you don't get flooded if there are alot of flaps, but it might be tough to get a single Message into the alert, I just grabbed the first result.

The text below is a variable that does a SWQL query (which you can use to pull any of the Syslog values by swapping them out.) Just out that in your email text or alert message and it Should work. Sadly with no data I am not able to test.

${N=Alerting;M=AlertName} was triggered.${N=SWQL;M=SELECT TOP 1 Message

FROM Orion.SysLog

Where DATETIME is not Null and MINUTEDIFF(DateTime, ToUTC(GETDATE())) < 5}

You can see that the interface trigger acctions is below, I hit 'Insert Variable' and checked 'Define SQL/SWQL Variable (Advanced)' and it basically pasted the text above into the page once I out my Query in and submitted.

jaybone

These are great, thank you!

While not specifically addressing the original problem (old .exe based alert doesn't appear to parse regex properly) it definitely produces the desired outcome. Probably just as well, since they've been moving away from those .exe tools anyway - I have to imagine they'll stop being supported at some point soon.

The statement I ended up with for the message variable had to mirror the trigger statement to get relevant log lines into the message (instead of just the last thing that came across). I can see where this would possibly grab the wrong info if there's a lot of flapping, but any trigger of this should warrant a more in-depth investigation by our network team anyway.

SELECT TOP 1 Message

FROM Orion.SysLog

Where DATETIME is not Null and MINUTEDIFF(DateTime, ToUTC(GETDATE())) < 2 and Message like '%flap%' and Message not like '%Gi3/0/23 and port Gi3/0/27%' and Message not like '%Gi3/0/27 and port Gi3/0/23%'

Thanks again!