Down alert received whilst it can still be ping

neilmborilla

I'd like to ask some guidance regarding the alert that we received. We are receiveing alerts from orion on a timely manner that the device has been down then up again for a couple of minutes. The thing is, it is not down becuase we can still ping continuosly the device. the device is a firewall and a DMZ.

any inputs are highly appreciated

ebradford

Hi. I did some more checking into the false positives we have been experiencing with our firewalls. I have found three causes, which I presume to fit the scenarios for different outtages, though two of them are the based upon the same idea.

The first two are regarding busy firewalls/routers: configuration and traffic. We recently had a reason to dramatically increase our firewall policies and logging for a short period. During this period, the firewall processor was busy processing traffic, and had to ignore lesser tasks, such as responding to SNMP. We received outtages during this period. It was strange, that we could see Ping reply, but SNMP kept timing out. Now, SNMP is a fairly low priority application, so it was being dropped. It became obvious that despite periods of normal traffic, that the firewall was too busy because of it's configuration.

Secondly, we have occasionally experienced false SNMP outtages at peak periods, as correlated with net flow data. So, you can see, two different reasons for the firewall being busy, same result - dropped SNMP requests.

Lastly, we sometimes get a few false alerts during non-busy times. What I have found in these is that the virtual firewall has jumped from one router to another, and maybe back again -- confusing the arp table. Some creedence was lent to this by deleting the arp table entry from the CMD prompt, and then the alert cleared. use: arp -d {ipaddress}

If your situation is similar to one of the above, what to do... in the case of confused arp entries, could attach a command to the alert which would run a command to delete the arp entry. For the busy router situations, you could perhaps increase the priority of SNMP, should your router allow for that, or you could try editingor re-ordering the policies to make work easier on the router. Remember, that a router will progress through policies until it finds the first matching policy, then it will execute and stop processing policies. So, it may behoove you to put the most used policies first (except the deny all) and the lesser used policies later. Netflow may help identify traffic flows, to help get the policies in order.

Anyway, I hope there's something you can use is all of that.