Rebuilding Solarwinds due to SUNBURST, etc.

Question

We're rebuilding the pollers to ensure nothing has crept in.   We have a primary poller and two APEs.  I just added a new APE and was intending to run it in Eval mode for 30 days whilst I transferred the agents over - our complex network has many firewall rules, I know it isn't going to quite as easy as select all agents on a poller and move to the new one - the old one is on-prem in a data center, the new one is in Azure.

My APE is showing as down in the All Settings->Pollers, but all services are up.  I moved 1x agent to it and when I glanced at the agent Application Templates were showing as Not Licensed.   Is that the reason it shows as down?

As we had to rebuild this due to SUNBURST I'm hoping Solarwinds will take this opportunity to give us a temporary license for 30-60 days - has anybody else had these issues when trying to move agents to new installation ?   I know it's going to take a few weeks to move all the agents.

jonathanswift · Answer

In /Orion/Admin/Details/Engines.aspx much later it is saying no database sync in 1113 minutes.

This is a new APE, new name, new IP, in Azure, to replace on-prem.  It was planned to build new pollers in Azure and migrate all the agents, we're decommissioning 4 data centers and moving everything to Azure.   If you install a standalone test system you get 30 days grace to test/use the product, but it seems like if you try to add an APE and hope for 30 days grace in the production environment it doesn't allow it.

Many of us have security teams asking if/how difficult it is to rebuild the software, replace the polling engines.  At the moment I'm planning to move 2 APEs out of data centers and in to Azure, by which I mean build brand new polling engines, new IPs, new names, then migrate the agents.  It will take weeks to move the agents - almost everything is agent-based.

These are new pollers, once the migration is complete we'll move the license key to the new pollers.

In dbo.Engines I see the new poller, type is additional, and the IP is correct.  Engine version is all ok - it's all the latest 2020.2.1 HF2.

New servers are all Windows 2019.  BusinessLayerPort is NULL, not 17777 like the others.  After installation the server was rebooted.

In Manage Nodes I can see the poller, that's how I moved 1 node to it for testing.  The node continues to say Not licensed.

BusinessLayerHost-SOLARWINDS.ORION.ANALYTICS.BUSINESSLAYER.DLL.txt says;

2021-01-16 13:45:35,028 [Main] WARN SolarWinds.BusinessLayerHost.BusinessLayerHostService - Check License had an exception: License is expired or not available
2021-01-16 13:45:35,033 [Main] INFO SolarWinds.BusinessLayerHost.BusinessLayerHostService - Switching to non-operational mode, waiting for valid license.
2021-01-16 13:45:35,038 [Main] INFO SolarWinds.BusinessLayerHost.PluginManager - Starting plugins on server configuration 'AdditionalPoller', HA standby mode 'NO'.
2021-01-16 13:45:36,243 [Main] INFO SolarWinds.BusinessLayerHost.PluginInstanceAppDomain - Plugin "Analytics Business Layer" doesn't support Polling Engine without valid licence and was skipped

I guess it won't allow it without a license - that's a big problem for those of us who have to rebuild pollers due to SUNBURST or just moving agents to new pollers.  You might be able to move agents really quickly with minimal downtime, I know I won't because there are a ton of firewall rules we'll discover along the way that need to be changed.  Security is rather tight.

cahunt · Answer

We might need a few extra details. First things first. I have had installs just were recently run successfully and everything was working, but the Polling Engine page, and/or the system health page was showing one of the pollers as down. In a few cases in the last 6 months, it took a little patience, as polling was there but we did not get the affirmed status on those Orion System Pages until after a few polling cycles. It has varied between a about 5-10 minutes to 30 or a little longer in some cases. But if not yet up....

When you say you are rebuilding the polling engines, does this include the primary?

Are we installing on new servers/rebuilt/same servers no change? And was the DB migrated to a new server or same instance of the db is still being used?

If you are migrating, either the same IP/Name or New you will need to migrate the license store. Check step 11 of the migration guide, under Step 2.

https://documentation.solarwinds.com/en/Success_Center/orionplatform/content/migrate_solarwinds_to_a_new_server_with_a_new_ip_and_hostname.htm

(about 1/2 way down the page)

As well, did you keep the engines table clean if we are using new server name/ip?  If you see extra lines in the Engines table then your EngineID reference for polling might need to be realigned as well. Check what is not polling and see if there is congruency on the assigned poller, and any other details noticed may be useful.

-CharlesH

Loop1 Systems: SolarWinds Training and Professional Services

* LinkedIN: Loop1 Systems
* Facebook: Loop1 Systems
* Twitter: @Loop1Systems