As a proud member of the THWACK MVP community, I felt it would be useful to share this story, so other community members will have a definitive answer to this question for their own security team/audits.
Last week, I had been going through some security woes as part of fixing a partially successful upgrade for a client. Upfront, the issue has been resolved, but the resolution has raised questions from my client's security team, so I'm sharing this here to help others who hit this when upgrading to the latest release.
Please note that this is NOT a criticism of the support I received, it was exemplary and I have said so in my feedback.
The pre-checks all passed, and it was only AFTER the upgrade was complete on the MPE (including configuration wizard run), when the configuration wizard step was being executed by Centralized Upgrade process on the APEs, that the problem reared it's head:
The configuration wizard on any of the Additional Polling Engines would not complete, complaining that they could not communicate with the Main Polling Engine's RabbitMQ:
I performed some manual port checks on each of the APEs back to the MPE on the RabbitMQ ports, which passed.. this was becoming a chin scratcher!
For reference, on each APE, I ran this command from an elevated PowerShell to confirm the TCP port was open::
Test-NetConnection -Computername YourMPEIPorHostnameHere -port 5671
This led me to dig deeper.
As it was getting late, and I needed this instance back up by 09:00, I raised a ticket with SolarWinds (Case #01547008) to look at this, as I was seeing some errors in the logs under the hood, indicating that the APEs didn't share a common protocol with the MPE, which was causing the error:
This got me thinking about the versions of TLS enabled between the MPE and APE potentially being the problem, as I have seen something similar in the past, I worked with support to find a resolution.
The issue appeared to be around crypto, a combo of TLS version mismatch and potentially cipher/hash/key exchange differences. The MPE was upgraded just fine.
Here's what the IIS crypto tool (a handy utility to have in your toolbox!) showed when the platform was broken, with the config wizards on the APE failing:
In the end, the solution from support was to download and run the aforementioned IIS Crypto too shown above (available from https://www.nartac.com/Products/IISCrypto/), and hit the 'Best Practices' button to restore any missing settings. This was done on the MPE and all APEs, and fixed the comms issues, and allowed the Configuration Wizard ran successfully on all six APEs. I had a working instance well before the client came in to work. Success?
Well, yes and no.
A subsequent call with the client and their security team, questioned why the vendor needed TLS 1.0 and TLS 1.1 enabled, as these are rightly seen as unsecure, especially where TLS 1.0 is concerned.
I had no answer, but said I would get back to them. Leading to this post.
For reference, this is what changed post the running of the tool:
- TLS 1.1 and 1.0 were restored to the server protocols, and TLS 1.0 was restored to client protocols.
- Triple DES 168 restored to ciphers.
- MD5 and SHA hashes restored.
- Diffie-Hellman ley exchanges restored.
I have reached out to the product owners with SolarWinds, and have asked that this thread is updated with the answer, so those who also come up against the same problem can be more selective than the 'Best Practice' button in the tool, but still end up with a working system.
