cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 16

Multiple alerts in a single alert cycle possible in Log Analyzer ?

Jump to solution

I was testing Log Analyzer using a script that sends out 7 unique syslog messages all at once.

The severity level (-l 1, -l 2, -l, 3, etc....) is unique for each message as well as the message text itself.

klog -h 10.157.148.152 -l 1 -m "shark level 1-Alert"

klog -h 10.157.148.152 -l 2 -m "shark level 2-Critical"

klog -h 10.157.148.152 -l 3 -m "shark level 3-Error"

klog -h 10.157.148.152 -l 4 -m "shark level 4-Warning"

klog -h 10.157.1481.52 -l 5 -m "shark level 5-Notice"

klog -h 10.157.148.152 -l 6 -m "shark level 6-Informational"

klog -h 10.157.148.152 -l 7 -m "shark level 7-Debug"

I then made a Log Analyzer rule that looked for the work 'Shark' in the message.

pastedImage_1.png

I then created an alert and checked the box below that states 'Create a new alert that fires whenever this rule triggers'.

pastedImage_2.png

Save the rule.

pastedImage_3.png

After that I went into the alert and the only change made to the one that Log Analyzer created was to add some values to the Message Displayed box, added an action to log to the NetPerfMon log and update Custom Properties with a URL.

pastedImage_4.png

When I run my script it fires off 7 syslog messages inside of a half second.

klog -h 10.157.148.152 -l 1 -m "shark level 1-Alert"

klog -h 10.157.148.152 -l 2 -m "shark level 2-Critical"

klog -h 10.157.148.152 -l 3 -m "shark level 3-Error"

klog -h 10.157.148.152 -l 4 -m "shark level 4-Warning"

klog -h 10.157.1481.52 -l 5 -m "shark level 5-Notice"

klog -h 10.157.148.152 -l 6 -m "shark level 6-Informational"

klog -h 10.157.148.152 -l 7 -m "shark level 7-Debug"

If I go into the new Syslog Viewer they look like this:

pastedImage_5.png

I would hope to see 7 syslog triggered alert messages show up in my All Active Alerts console but the alert is only catching one of them.

pastedImage_6.png

Once this alert is acknowledged it will never trigger again regardless of how many syslogs are sent until the Orion server is rebooted.

Seems like I have something set up wrong?

0 Kudos
1 Solution

Changing those values after the fact will not fix it if the alert was already active.  Change them to trigger each time the condition is met, clear out any currently active instances of the alert.

The way that LM stacks alert events up on the node is a bit problematic for alerts like this, you will always only have a single alert active at once for each combination of LA rule/node.  Tthe others triggered and cleared pretty much instantly.  The "best" way I have found so far to work around the issue is to create several alert conditions, so you would need to set up rules in LM for each severity or each type of alert that you want to trigger so they don't conflict with or over write each other.

- Marc Netterfield, Github

View solution in original post

5 Replies

Changing those values after the fact will not fix it if the alert was already active.  Change them to trigger each time the condition is met, clear out any currently active instances of the alert.

The way that LM stacks alert events up on the node is a bit problematic for alerts like this, you will always only have a single alert active at once for each combination of LA rule/node.  Tthe others triggered and cleared pretty much instantly.  The "best" way I have found so far to work around the issue is to create several alert conditions, so you would need to set up rules in LM for each severity or each type of alert that you want to trigger so they don't conflict with or over write each other.

- Marc Netterfield, Github

View solution in original post

Level 16

Worked with Support on this today and mesverrum Expert is correct in his answer.

You can only have one alert trigger, per node, per rule. Until that one is reset no others will show up in the All Active Alert console. 

Level 16

Thanks! mesverrum​  I'm still new to the product and working through how the alerts are going to work. We have a lot of hosts that send syslogs or traps in rapid succession such as whenever someone swipes an entry badge.

I will have to get creative.

0 Kudos
Level 14

What is the reset condition on this alert? Memory is telling me that it might default to Reset automatically after 60 minutes, but you likely want to set it to "No reset condition - Trigger this alert each time the trigger condition is met".

0 Kudos
Level 16

There are a couple reset options, I tried both with the same results.

It appears that the alert is only grabbing the first event that happens within the alert cycle and skipping the rest. 

pastedImage_0.png

0 Kudos