2023.3.0 upgrade fun.

Hi folks,

OK, so I've been trying to upgrade our 2020.2.6 installation for a month, going up to 2023.2.1. Various delays and the occasional customer disaster meant that this didn't happen until yesterday, and in the meanwhile Solarwinds had GA'd 2023.3.0 so I thought why not. Centralised update wasn't possible because of various faults (and a SW tech said 'don't') so offline update it was.

To my surprise, the primary poller upgraded ALMOST successfully, just a couple of hiccups that had already been reported elsewhere so were easily fixed in the database. Primary poller was back up less than 2 hours after starting. Splendid. I then did the secondary pollers (2) and the additional web server - we have this because when SW was on prem the primary poller was swamped so we migrated all the web traffic to an AWS and there it stayed even once we'd migrated to Azure in 2021. IIS on the primary poller is disabled.

Oh yeah, all this is in Azure behind a Traffic Manager, an Application Gateway and a pair of firewalls.

Webserver upgraded fine, config wizard ran fine and I can log in locally, all this is good.

What I CAN'T do is log in from outside the platform, ie beyond the Azure App Gateway. The requests seem to go down a black hole and never appear on the AWS in the logs. Strangely, the probe page does work, so swMonitoringURL/.../probe.html works fine and the app gateway backend health is happy. However, any request to swMonitoringURL just disappears and eventually the gateway times out.

Through all of this, the only thing that's changed is the Solarwinds binaries on the AWS and the website itself in IIS. Spent 2.5 hours last night with a SW tech (case 01420612) to no avail. They're still examining logs.

Has anyone seen this behaviour before?

Cheers

Adrian

Find more posts tagged with

Comments

adam.beedell

Super weird one, and I dont have the azure bits for any equivalencies. There's some port stuff with 17777/17778 changing toward 17774 that jump to mind perhaps, though not sure how that'd be involved exactly.

Very curious how it's solved as you continue

fop

I don't think many people have the Azure bits unfortunately. The 17774 port stuff is part of SWIS, and the current ports also still work but will be deprecated in a future release. If it was port related though then it wouldn't work at all, and the fact that it does but only for browsers with direct access to the private IP of the AWS is causing SW and Microsoft to have a good head scratch.

That the probe page is accessible from anywhere, even through the app gateway, is bonkers. It's an instant response too, so why isn't /orion/login.aspx also instant? On the odd occasion it DOES work, you can go through the MFA and then... nothing. Nada. Not even a timeout, just a not-progressing status bar (in Safari).

We upgraded the logging in IIS this morning to also show where external requests are coming from and can see our testing workstations are being sent the login.aspx page with all the css it needs. One of our firewall techs is wondering if SW have added some sort of man-in-the-middle detection, because packets from the app gateway are modified to have the address of the gateway as their source. I can't see any evidence of this in the IIS settings though.

I'm supposed to be on holiday today. Sigh.

fop

OK, it's fixed. 2 whole days plus hours in meetings with Microsoft and SW techs.

Problem was that previous generation SW webpages identified only as 'web browsing' so were allowed through the firewall. Today with Microsoft 3rd line they told us which packets were being dumped by our firewalls but couldn't tell us why so we dug deeper.

Turns out that from a version later than 2020.2.6 the web page identification changed to be things like 'solarwinds-base' or 'solarwinds-sam' or 'solarwinds-npm' and our firewalls were dumping them because they weren't recognised.

This might not be news to anybody else but it sure was news to us We never thought of checking the firewalls for that because they hadn't changed, only the website had. 5 minute fix.

adam.beedell

This probably will string someone then, good troubleshooting

fop

I am now very much on holiday for the rest of the week, he says checking emails. Looking forward to explaining all this in the outage report

charlesjsmith

Hi Adrian.

Thanks for sharing this information. I'm planning to do similar in the near future and i am running at the same version you were and planning to go to the same release you did. If you don't mind sharing, i'm wondering what the hiccups were that you mentioned " just a couple of hiccups that had already been reported elsewhere so were easily fixed in the database".

Thank you!

fop

Hi Charles,

The first issue was simply that I'd left the backup dialogue box open on SQL Studio after doing Yet Another Database Backup before starting. That was enough to lock the DB so I got this:

https://support.solarwinds.com/SuccessCenter/s/article/Error-Backup-and-file-manipulation-operations-such-as-ALTER-DATABASE-ADD-FILE-on-a-database-must-be-serialized?language=en_US

The only thing after that was

https://support.solarwinds.com/SuccessCenter/s/article/Error-while-executing-script-There-is-already-an-object-named-CLM-CloudJobSettings-in-the-database?language=en_US

The firewall issue was a doozy though, because we filter traffic based on identified application and the new solarwinds pages weren't simply web pages any more.

Cheers