Trap to Alert - Fortigate FWs

Preamble:
We're developing an alert from a Fortigate firewall trap. The specific one at this stage (though it is up for discussion) is: fgTrapHASwitch

Initially to ensure the process worked I also enabled the logic of the trap rule to look for:

OR Contains fgTrapHASwitch
OR Contains fgTrapHaStateChange
OR Contains fgTrapHaHBFail
OR Contains fgTrapHaMemberDown

Once I knew it was working, I reduced it to just the HASwitch.

Only issue at this stage is, and it's probably a Fortigate thing, is that it generates 4 HASwitch traps in the space of 10m. So Solarwinds sees the 4 traps and raised 4 alerts.

Question:
Is there a way, within the alert logic or something I haven't seen to only raise the one alert and to NOT raise more for the next x minutes?

Find more posts tagged with

Alert

fortigate

firewall

trap

Accepted answers

stuartd

For the benefit of anybody else that sees this thread I finally found a solution to this.

The short version is:

Despite contacting and asking Fortigate and being told they don't have an OID that monitors failover, we have found that they do and it is:

FgHaStatsSyncStatus 1.3.6.1.4.1.12356.101.13.2.1.1.12

The output of this is 1 or 2 or 3

1 indicates master unit
2 indicates standby unit and
3 indicates a standalone firewall

So there was/is one of two ways to do this, and that I guess all depends on how comfortable you are with SWQL, etc.

Option 1: create 2 UnDPs on the above OID and name appropriately as Master and Standby. You need to set this to a Get Next. You then also need to apply all your master/primary firewalls to the master UnDP and all the standby/secondary ones to the other UnDP.

In your alert logic, when assigning the UnDP, you need to set it to say: if value does not equal 1 (or 2) as appropriate.

Option 2: Create a single alert using Custom SWQL, with full details of how in this thread: https://thwack.solarwinds.com/t5/NPM-Discussions/Trigger-Conditions-with-UnDPs/m-p/618304#M152598

All comments

KMSigma.SWI

<Preamble> I don't have a Fortigate on which to do any of this testing, nor am I a fan of using traps if there are other options available. I find them too chatty and obtuse.</Preamble>

Do you happen to know if it will ALWAYS send 4 traps? Because you could use that as part of your logic. In Log Analyzer, I would do something like this for the conditions:

Then just make sure that the "Send a Log Rule Fired event to Orion Alerting" and "Create a new alert that firews whenever this rule triggers" are checked.

Then you can just edit the alert in the Alert Management and include the things about the trigger details and any additional filters.

stuartd

Kevin,

Thanks for the reply. I should have given more detail but have found in the past when I do that I don't get any replies so deliberately tried to keep my initial request short.

I hear you about the traps, but in some cases we have no other option as a bunch of our virtual Fortigates have been deployed with a single management IP, so we have to use what we can. It also didn't help that Fortigate Tech Support told us that there was no OID for HA events, but then we went and found there is

So first answer: no, at this stage I don't know if they always send 4 traps as we have to wait on a Fortigate to failover to do any testing. Having looked in to this specific 'failover', I can now see that only one of the two ports in the HA group 'failed' and did so twice within about 5/10m. This could have bene as simple as the switch in the middle being rebooted, or a dodgy cable...but in truth wasn't a true failover as the heartbeat was never lost. The takeaway though is that we would still raise an incident to investigate what we can.

Anyway, the trap side of it I have worked out and is along the format your screenshots show. I can share those at a later stage once I'm happy with the whole trap to trigger to alert to incident process.

The real question is/was, how can I, from within the trigger alert logic only fire an incident from the first of the 4 in that 5/10m window?

If it isn't possible, which I suspect is probably the case, then we would simply close the others off as duplicates.

KMSigma.SWI

It's a real shame that I can't see into their API documentation, because that's how I'd try to do it.

I'm honestly falling back on SWQL because it's my crutch and I do many, many things there. This is what I came up with. You can test the logic by going to YourOrionServer.Domain.Local/.../swis.aspx, select Orion.Nodes from the entity drop-down and paste the below query in replacing whatever is in the text box.

SELECT Nodes.Uri, Nodes.DisplayName
FROM Orion.Nodes AS Nodes
----  Lower Portion ----
INNER JOIN Orion.OLM.LogEntry AS [LogEntry]
  ON [Nodes].NodeID = [LogEntry].NodeID
WHERE [LogEntry].LogMessageSource.Vendor = 'Cisco' -- 'Fortigate' (or whatever)
  AND [LogEntry].LogType.Name = 'Traps' -- 'Syslog', 'Traps', 'Windows Events', 'VMware Events', 'Log Files'
  AND MinuteDiff( DateTrunc( 'minute', [LogEntry].[DateTime] ), GetUtcDate() ) <= 60 -- number of minutes between checks
--           === Trap Filter this block or the one below ===
--           === RECOMMENDED: By TrapType Name           ===
  AND [LogEntry].FieldValues.Name = 'TrapType'
  AND [LogEntry].FieldValues.TextValue LIKE 'SNMPv2-MIB%' --'fgTrapHa%' 
--           === Secondary Option: By OID Values         ===
--  AND [LogEntry].FieldValues.Name = 'TrapOid'
--  AND [LogEntry].FieldValues.TextValue IN ( '1.3.6.1.4.1.12356.101.2.0.401', '1.3.6.1.4.1.12356.101.2.0.402', '1.3.6.1.4.1.12356.101.2.0.403', '1.3.6.1.4.1.12356.101.2.0.404', '1.3.6.1.4.1.12356.101.2.0.405' )

In my example above, I needed to use Cisco, an SNMPv2-MIB trap type, and a 60-minute timeframe. Tested on Orion Platform: 2020.2.1 HF1.

I've also attached my best guess at a functional alert for you. Like I said, I can't test this in a lab, much of this is guesswork and will require tweaks for your system.

I'm tagging in @pravs88 who is the Product Manager for Log Analyzer who might have a better way to do this or know of another solution.

attachments.zip

stuartd

For the benefit of anybody else that sees this thread I finally found a solution to this.

The short version is:

Despite contacting and asking Fortigate and being told they don't have an OID that monitors failover, we have found that they do and it is:

FgHaStatsSyncStatus 1.3.6.1.4.1.12356.101.13.2.1.1.12

The output of this is 1 or 2 or 3

1 indicates master unit
2 indicates standby unit and
3 indicates a standalone firewall

So there was/is one of two ways to do this, and that I guess all depends on how comfortable you are with SWQL, etc.