HA FAILOVER INTERFACE COLLECTION DIED

Question

We did our first production fail over test and all looked well and I was about to go to sleep when i noticed interface collection stats was dead.

We lost 4 hours of stats and now our PMs are going to go nuclear and come after me with pitchforks on the first of the month.

What can cause Interface stats to die after failing back? Anyone else ever experience this?

hpstech · Answer

We had to uninstall all Orion services and re-install. I think the Job Scheduler was the main culprit. The uninstall and reinstall of all services is the #1 solution for many problems.

It makes me nervous re-enabling HA to be honest.

I have 20 project managers who bark if stat collection dies. If this happens once a month with unplanned fail-overs, I'll be looking for a new job.

Management is having me demo Logic Monitor today at 2PM. I know they won't compare to SolarWinds but I have to follow orders and look for a SAAS based monitoring solution because the perception now is SolarWinds is too much of a management challenge and breaks too often. We're providing a service to many huge companies and have little margin and tolerance for error.

As much as I try to keep people happy with SolarWinds, I cannot deflect these perceptions when situations such as this thread happen, and many others. I know many people can relate. SolarWinds is awesome but with so many moving pieces and parts, it contributes to stability issues that nag us. Our NCM jobs hang up at least once every 2 to 4 weeks, and cause a cascade of other symptoms, the main one being loss of statistical collection which kills us. Our compliance manager sees me asking for reboots once a month to fix SolarWinds, and this contributes to the SolarWinds reliability questions. There is nothing more I hate than putting in change requests for SolarWinds reboots.

Oh how I wish it was simple, but we are light years from a simple SAAS monitoring platform that can do even 10% of what SolarWinds does. We all know that.

aLTeReGo · Answer

Without some diagnostics or logs its difficult to say for certain, but if I had to make a guess it would be that the interfaces plugin to the Collector is missing or corrupt on the second member of the HA pool. I would recommend opening a case with support so we can identify the cause.

o.rodriguez · Answer

Hi hpstech

We are having a similar issue but with VNQM.

Once we perfom a failover on a APE Pool, this stop collecting stat for all IPSLA operations related to that PE, and a short time later, al VNQM stats recolection stops in all three PE.

Interfaces, Nodes and everything else works perfect during the event, just Voip Operation Results stops being collected

This back to life even doing nothing, it just starts working again by itself.

Of course, we currently have an open case but we still have no resolution. If it is solved I will put it here