Is it possible to create an alert that only triggers after multiple unique syslog events occur?

dswift01

We had an interesting outage earlier today, and as a result i was tasked with trying to create a new alert. There was a circuit that did not go hard down, however traffic was unable to pass through it. Because of how it was configured, the parameters for failover to the secondary circuit were not met. This alert is going to be more or less of a CYA type thing as they are also planning to fix the failover setup, and want this still as a backup. Will try to explain as best as I can without giving company info.

We are using FortiGate FG200 and FG600 devices as "SD-WAN routers" using 2x at each site in an active/standby HA setup.

Each circuit is split into 2 paths, one for MPLS and one for DIA

There are 3x IP SLA pings setup per path.

SolarWinds receives a syslog notification each time there is an event

There is a separate message sent per IP SLA ping, meaning if 01, 02, and 03 all failed, 3x syslog messages would be sent

I was asked to create an alert that would only fire off when all 3 of IP SLAs for a single path are "dead" at the same time. What makes this more fun is that all sites/FortiGates are configured with the same generic verbiage for the IP SLA naming, and all hit the same IP addresses to test.

TL;DR version - Is is possible to setup an alert that only fires off when 3x syslog events happen? I was thinking it could be done using complex conditions. Then realized i may need to create a separate alert per location, or separate syslog rules per location, but not sure.

Hoping someone else has had to do something like this in the past and can share some advice, it would be greatly appreciated.

vinay.by

Rather than a complex condition I think this can be achieved by having separate syslog processing rules/tags. If you have 3 different processing rules each specific to one scenario you can use them with an AND clause on your alert. As well look at Evaluate Frequency of the Alert - the default is 1 min right, so in your case if these 3 syslogs might come in within 2 mins or 5 mins adjust your "Evaluate Frequency of the Alert" to 2 mins or 5 mins accordingly. There isn't really a need to check this every 1 min unless required.