Palo Alto Volume no longer exists, Interface no longer exists, Data collection suspended/terminated. How to alert on these?

Question

Greetings Thwack community!

Recently I had a Palo Alto have a problem with it's card in slot3.  SolarWinds logged a set of events:

3/20/2021 21:38wdc-ent-fw01-Slot-3 Data Processor-0 Software Packet Buffers- Volume has reappeared. Data collection resumed.3/20/2021 21:38wdc-ent-fw01-Slot-3 Data Processor-1 Packet Descriptors- Volume has reappeared. Data collection resumed.3/20/2021 21:38wdc-ent-fw01-Slot-3 Data Processor-0 Hardware Packet Buffers- Volume has reappeared. Data collection resumed.3/20/2021 21:38wdc-ent-fw01-Slot-3 Data Processor-0 Packet Descriptors- Volume has reappeared. Data collection resumed.3/20/2021 21:38wdc-ent-fw01-Slot-3 Data Processor-1 Software Packet Buffers- Volume has reappeared. Data collection resumed.3/20/2021 20:47wdc-ent-fw01-Slot-3 Data Processor-2 Packet Descriptors- Volume has reappeared. Data collection resumed.3/20/2021 20:47wdc-ent-fw01-Slot-3 Data Processor-2 Hardware Packet Buffers- Volume has reappeared. Data collection resumed.3/20/2021 20:47wdc-ent-fw01-Slot-3 Data Processor-2 Software Packet Buffers- Volume has reappeared. Data collection resumed.3/20/2021 20:24wdc-ent-fw01 - ethernet3/27 · Old Slot 4 Port 25 Interface restored. Data collection restarted.3/20/2021 20:24wdc-ent-fw01 - ethernet3/26 Interface restored. Data collection restarted.3/20/2021 20:24wdc-ent-fw01 - ethernet3/28 · Old Slot 4 Port 26 Interface restored. Data collection restarted.3/20/2021 20:24wdc-ent-fw01 - ethernet3/25 Interface restored. Data collection restarted.3/20/2021 20:10wdc-ent-fw01 - ae3 · Trust Up3/20/2021 20:08wdc-ent-fw01 - ae4 · Trust Up3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-1 Packet Descriptors- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-0 Hardware Packet Buffers- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-0 Packet Descriptors- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-1 Software Packet Buffers- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-1 Hardware Packet Buffers- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01-Slot-3 Data Processor-0 Software Packet Buffers- Volume no longer exists. Data collection suspended.3/20/2021 20:04wdc-ent-fw01 - ethernet3/27 · Old Slot 4 Port 25 (index: 224) Interface no longer exists. Data collection terminated.3/20/2021 20:04wdc-ent-fw01 - ethernet3/28 · Old Slot 4 Port 26 (index: 225) Interface no longer exists. Data collection terminated.3/20/2021 20:04wdc-ent-fw01 - ethernet3/26 (index: 223) Interface no longer exists. Data collection terminated.3/20/2021 20:04wdc-ent-fw01 - ae4 · Trust Down3/20/2021 20:04wdc-ent-fw01 - ae3 · Trust Down3/20/2021 20:04wdc-ent-fw01 - ethernet3/25 (index: 222) Interface no longer exists. Data collection terminated.3/20/2021 20:03wdc-ent-fw01-Slot-3 Data Processor-2 Packet Descriptors- Volume no longer exists. Data collection suspended.3/20/2021 20:03wdc-ent-fw01-Slot-3 Data Processor-2 Hardware Packet Buffers- Volume no longer exists. Data collection suspended.3/20/2021 20:03wdc-ent-fw01-Slot-3 Data Processor-2 Software Packet Buffers- Volume no longer exists. Data collection suspended.

SolarWinds managed to find nothing to generate an alert on, despite successfully logging events.  Notice the interfaces involved were not down, the volumes did not generate errors, and in addition to these issues, the panSysHAState changed.

The good news here is that a failover happened at the start of the strong of events, so there was minimal impact to the users.  However for an hour and a half this firewall was inoperable, and SolarWinds saw 30 events, so there is opportunity here to catch thse the next time they happen.  So what do we do to make that happen?

The first thing I have done so far is to capture changes to the panSysHAState oid.  The way I did that was to create a custom property to store the HAState in, then use an alert to compare panSysHAState to the toe custom property.  Then after the alert is dispatched, let some time pass, then store the new state in the custom property.  So during an ha failover, notifications go our to get people to look at the firewall.

I saw another thread where someone asked for help dealing with events like this, and they were told to use "volume down" and "interface down" alerts.  I would like to point out that no interfaces went down, no volumes went down.

When I see these "resource no longer exists.  Data collection terminated/suspended." alerts, what do I even alert on to catch them?  What OIDs are these coming from?  is there a way to get details of an event so that we know how to build an alert for it?

Keoki

mesverrum · Answer

support.solarwinds.com/.../List-of-NPM-NTA-VNQM-MIBs-and-OIDs-used-for-Polling

keoki · Answer

Yeah, HA alerting is working really well.

So I think I can write an alert for the disappearing interfaces, but I need to know what OID is being used to poll these.

How do I find out what OID is used for a specific event in the event logs?  If I knew that I would be able to handle any logged event.   For my custom pollers I have full visibility and control.  But for things Solarwinds detects out of the box, I seem to have absolutely zero visibility.  So for "well supported" oid's, I feel unsupported.  I'm hoping that is because I just don't know how to look up what SolarWinds knows how to poll by default.  I'm hoping they don't hide that info, but the last time I asked in a support ticket, no answer was offered.

Seashore · Answer

Sorry, I found hardware health about power, fans, temp and disk but no overall or other hardware status.