High-Availability/Failover Issue with Node/SAM Polling following Failover

t.m-k

Hi THWACK,

> Running a HA deployment

> Just upgraded to 2025.1.1

> Failed over from Main/Primary (MPE1) to Standby (MPE2)

> Failover itself was fine and seamless

> The issue which followed: For the ~2000 nodes now assigned to MPE2 (MPE2 took over node monitoring from MPE1) - following node polling no new node statuses seem to be collected after the failover, and Components within SAM templates errored with 'Timeouts'

> This attributed to ~250+ alerts

> A weird and extremely long error occurred when visiting the License Manager page

> After click 'Synchronize' the error disappeared and did not show up again when navigating away and back to the License Manager page

> This did not resolve the issue neither did a reboot of MPE2

> After failing back to MPE1, all issues were resolved with polling and alerts auto-resolved/cleared themselves

Has anyone ever experienced this behaviour following a failover? Any ideas, THWACK?

Thanks for your time.

--t.m-k

marcrobinson

Check the orion logs, I think it might be the orion module business layer and maybe core business layer on the failover MPE. There are several and I am not looking at a server right now. Look for anything that is large - like 10 mb or more. As solarwinds keeps the last five or so around, you can filter by name and date, in addtion to size. They are appended with a number at the end. Notepad will generally do just fine to review them. Also - are the nodes agent based? If not, and the logs look fine, the usual involving firewall hijinks etc. I suspect firewalls are not your problem though. 

Also check the following:

DB connection from the failover MPE - make sure it can connect. Look at orion.engines entity for keepalive for example - gives a timestamp. 

This could involve the rabbitmq or sqllite stuff for under the hood processing. - grab diags and open a ticket.