Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Trigger Conditions with UnDPs

I think I have mostly worked out how to alert on an UnDP.

Set the scene: I have created a UnDP based on the OID: 1.3.6.1.4.1.12356.101.3.2.1.1.4 which comes back correctly as 'fgVdEntHaState'. I have further tested and assigned this to a bunch of our Fortigates.

I get 3 results 1, 2 or 3. A perfunctory glance tells me this:

1= active unit
2 = standby unit
3 = standalone unit and subsequently I have unassigned these nodes.

Additionally I know that 1 = Active / Primary as this works with WhatsUP Gold (our former monitoring tool).

The Logic:
My trigger conditions are:

The premise being here, that if the device HA fails then it changes to 2 and thus matches the alert, so would trigger. The last line is just utilising a custom property we have so it only looks at kit that is live.

The question:

So given all the above, then let's say we have 10 Fortigate FWs in HA pairs that would mean 5=1 and 5=2 - so surely when I get to the summary page, the little popup box should alert me to the fact that this would immediately trigger on x objects. Only it is stubbornly saying 0.

Any ideas why? And if not, then any ideas on a better way to look at this OID table and get its output to be alerted on?

Find more posts tagged with

Accepted answers

stuartd

I have raised a support case and will await their response, but in the meantime I have continued to think on this. For now, and longer term it will mean more work and/or having to remember to add in new firewalls as they come on board, but it is the best I can find

So...

set up the UnDP using default values but change it to a get next
set up trigger condition in the following manner:
Assign all the UnDP to all the Master / Primary devices then ...
Duplicate the UnDP and change the name to Fortigate_HA_Standby
Unassign current nodes from it and assign all secondary/slave/standby devices to it
Duplicate the alert but change the 'Custom Node Poller' line to is not equal to 2

This gives us the best of both worlds in that we monitor both sides for the failover.

If support come back with a way to do this as a single alert then I'll update, but meanwhile I'll accept this as the solution.

Thanks @dpad and @Seashore I owe both of you a virtual pint/coffeee/drink of choice.

Seashore

@stuartd hold that Pint, I might have a better solution!!! Couldn't let this go and what better end to a Friday than to create a nice SWQL-query.

Create a new alert rule, a "Custom SWQL Alert"
Choose "Custom Node Poller" as target.
The first part is "read only" and can't be changed, but the rest we can alter. Add this:

INNER JOIN
(
SELECT
CustomPollerAssignmentID
, Count(DISTINCT RawStatus) AS Statuses
FROM Orion.NPM.CustomPollerStatistics
where
Datetime>Addminute(-30,GETUTCDATE())
GROUP BY CustomPollerAssignmentID
) AS History
ON CustomPollerAssignmentOnNode.CustomPollerAssignmentID=History.CustomPollerAssignmentID

WHERE
CustomPollerAssignmentOnNode.CustomPollername='haCurState' -- Change to your UnDP's name
AND History.Statuses>1

So this will alert IF a Node Custom Poller, in above example named 'haCurState', has more than 1 different states last 30 min, it has changed state.

Change the name of the CustomPoller and give it a try. Have just tried it in SWQL-studio but it feels like it should work.

Please try it out and see if it works IRL also.

Peace!

All comments

dpad

Hello @stuartd

I replicated your alert and it's working fine in our environment.

Created custom property where I placed cluster, active and standby IP/Devices.

UnDP values on devices are:

Cluster (VIP) = 1, Active = 1 and Standby = 2

Do you have any other settings on the Scope? How about the UnDP properties? The OID default state is "Get Next".

I attached a couple of screenshots for reference.

Thanks!

attachments.zip

stuartd

@dpad your screenshots didn't come through - it tells me they aren't available. You can copy & paste them direct into the reply if that helps.

That said, if I understand you correctly, you said this is working for you. Have you been able to test a failover??

Seashore

You are right, half the nodes should have an alert on them if it would work.

I would take out the SWQL from the alert and test it on SWQL-studio.

Press the little arrow to the left on the yellow bar and press "show swql".

stuartd

Thanks @Seashore I knew that was there but hadn't made that leap in logic to copy and paste that to SWQL Studio.

The code becomes:

SELECT E0.[Uri], E0.[DisplayName]
FROM Orion.NPM.CustomPollerAssignmentOnNode AS E0 
WHERE ( ( E0.[CustomPoller].[UniqueName] = 'fgVdEntHaState' ) 
AND ( E0.[CustomPollerStatusScalar].[RawStatus] != '1' ) 
AND ( E0.[Node].[CustomProperties].[Node_In_BAU] = 'True' ) )

So, there's something about that 2nd AND statement that doesn't generate any content, just the column headings.

It's a start to play and see what I'm missing.

stuartd

And found it ...

I've changed the UnDP to a get next as pointed out by @dpad above but somehow I overlooked that. Now I have 103 items in scope that would be triggered - now whilst that number seems high (it is) at least I'm moving forward.

stuartd

@Seashore @dpad

OK - so I have this working, but I'm still concerned about the logic.

What we really need to know is when/if the value changes on any of the devices from 1 to 2 or from 2 to 1.

Any ideas on how I achieve that? Do I need to set the UnDP to rate? And if so, do I need to worry about the other boxes?

Seashore

Thought you might come to that...so I took a look but I did't find any obvious solution. I'm afraid this might need a custom swql to solve. To look at the history and comparing the values. Probably doable but nothing I have in my library.

dpad

@stuartd

The default value type for this oid is 'raw value"

I don't really use this OID for alerting so I haven't tested it on actual failover.

The setup I have for failover scenario is using the event trigger alert "System Name has changed" and making sure to only alert on the "Cluster IPs" and not on the mgmt IP of the Active and Standby FWs.

This way when the cluster IP name changed then it would mean a failover occurs.

You can set up other alerts for the different status which is on the same table entry on this MIB/OID group.

HA Sync status:

FgHaStatsSyncStatus 1.3.6.1.4.1.12356.101.13.2.1.1.12
Integer values:
unsynchronized(0), synchronized(1)

It seems you already have active/standby IP monitored so you can create a Node down alert for each device on the cluster supplementing the HA sync status.

Polling 3 IPs in total:

Cluster IP = alert for Name change, HA sync status and Node down
Active/Standby IP = Node down (using a different custom property)

This should cover most incidents on the devices.

Thanks!

stuartd

OK thank you again - I will raise a case I think.

stuartd

Again the reply is appreciated...

The problem we have is that a lot of our Forti's are virtual and I don't know if that causes an issue, but whomever set them up decided not to set up individual management and a cluster IP so the name change option won't work in these instances.

As I mentioned to @Seashore I'm going to raise a case and will report back if I get any success.

But in thinking about it, I could always set the logic to be: 'if UnDP value does not = 1 then alert' - yes it will mean a bunch of devices alert initially, but then we will always be notified of a primary failing over - as I'm sure you are aware, not every HA event means loss of node so just monitoring nodes won't work either.

stuartd

So...

set up the UnDP using default values but change it to a get next
set up trigger condition in the following manner:
Assign all the UnDP to all the Master / Primary devices then ...
Duplicate the UnDP and change the name to Fortigate_HA_Standby
Unassign current nodes from it and assign all secondary/slave/standby devices to it
Duplicate the alert but change the 'Custom Node Poller' line to is not equal to 2

This gives us the best of both worlds in that we monitor both sides for the failover.

If support come back with a way to do this as a single alert then I'll update, but meanwhile I'll accept this as the solution.

Thanks @dpad and @Seashore I owe both of you a virtual pint/coffeee/drink of choice.

Seashore

@stuartd hold that Pint, I might have a better solution!!! Couldn't let this go and what better end to a Friday than to create a nice SWQL-query.

Create a new alert rule, a "Custom SWQL Alert"
Choose "Custom Node Poller" as target.
The first part is "read only" and can't be changed, but the rest we can alter. Add this:

WHERE
CustomPollerAssignmentOnNode.CustomPollername='haCurState' -- Change to your UnDP's name
AND History.Statuses>1

So this will alert IF a Node Custom Poller, in above example named 'haCurState', has more than 1 different states last 30 min, it has changed state.

Change the name of the CustomPoller and give it a try. Have just tried it in SWQL-studio but it feels like it should work.

Please try it out and see if it works IRL also.

Peace!

stuartd

@Seashore now that looks intriguing, and longer term is a **bleep** site easier to manage then having my current workaround.

I can see how that SWQL works, just I know I never would have been able to create it. I will give it a try sometime and report back.

Diolch yn fawr.

stuartd

OK - so I've built the alert now but left it that it only eMails me.

If/when I get an eMail that should also correlate with the dual alert functionality I've already deployed. Once I know it's working then we can reduce to just the new alert. And would certainly ease ongoing support for when new Fortigates come onboard - we just add them to the one UnDP, rather than having to work out which is master / standby.