When you have to change all passwords for the Orion monitoring platform

File under #2020:
Situation

You are required to change all Orion-related passwords in response to a security breach or some other event. The affected credentials may include the following list, and possibly more:

• A domain-based service account used for the Orion platform's Windows services*
• A domain-based account used to access the SQL database (or a SQL-based login)
• Domain-based accounts used for monitoring Windows servers and applications with WMI
• A domain-based account with an Exchange Server mailbox for sending alerts
• SNMP community strings and/or SNMPv3 credentials
• Application credentials
• Hypervisor credentials in NPM, VMAN, or SAM
• SSH credentials in NCM

This write-up will focus on the first four types of accounts, which should all be domain-based accounts in Active Directory, unless you are running Orion services as the default NETWORK SERVICE security principle. 

* You may find that your Orion services are running as NETWORK SERVICE instead of a service account. Per SolarWinds documentation, this is a supported configuration.

Solution

Let's skip to the spoiler: in this kind of scenario, the least impactful and most secure approach is to create new domain-based accounts instead of changing passwords.

Step 1: Orion Service Account

• Create a new service account for Orion in Active Directory and grant it the appropriate permissions on your Orion servers.(Or let them just run as NETWORK SERVICE, if that fits your environment's requirements.)
• Coordinate with your DBAs to grant this new service account the required permissions on your Orion database[s].
• Run the configuration wizard on each of your Orion servers, providing the new credentials for the services and for the connection to the SQL database[s].
• Once you have confirmed that the Orion services are working properly, the original service account should be disabled.

Step 2: WMI Monitoring Accounts

• In each Active Directory forest that you monitor, it is best to create at least 3 accounts for monitoring Windows servers:
  1. One for monitoring your domain controllers (regardless of which template you use).
  2. One for monitoring your Exchange servers with AppInsight for Exchange.
  3. One for monitoring all other member servers.
  4. Also: use a dedicated account for monitoring your SharePoint farms (if you have SharePoint) and a dedicated account for the AppInsight for SQL Server template, as both of these require special permissions for the service account.
• Add these new WMI account credentials to Orion.
• Assign these new WMI credentials to the relevant nodes.
• Once you have confirmed that nothing else is using your old WMI monitoring accounts, they should be disabled and removed from any privileged groups.

Step 3: Mailbox Accounts

How are your alert emails and job notification emails sent from Orion? If you have created an Exchange mailbox for your original service account to send alerts, here is another best practice to consider: use a separate account with its own mailbox. When using the same account for email and for the application itself, you increase your risk of an accidental account lockout bringing down the whole application. This has potential to happen any time the SMTP settings are changed for the Orion platform, and any time an NCM job's email notifications are set.
Does this one require a brand new account and brand new mailbox? I'll leave that up to you and your security team. A password reset may be deemed sufficient. If new accounts are mandated across the board, then be prepared to help recreate any inbox rules that your alert recipients have configured in their mail clients.

Step 4: Final Validation and Cleanup

These final steps may vary depending on how your environment is configured and what your policies around account lifecycle require.

• Does anything else utilize your WMI monitoring accounts for similar purposes? It would be best to not share these accounts, but use credentials specific to each platform. Work with your Active Directory team and/or Security team to audit domain controller logs and make sure that these accounts are not still being used by anything else.
• What is your account lifecycle policy? You may be required to disable but keep old accounts for security purposes. Check any safeguards that are in place to make sure that these cannot be re-enabled and re-used, or delete these accounts after confirming that they are no longer in use anywhere.

Q: Why shouldn't we just change the passwords?

This can certainly be done, but there is actually more work involved, more potential disruption, and ir provides less guarantee of mitigation if any of your accounts were ever compromised. At a very high level, the process for this approach would include the following steps.

(Temporarily stopping the use of WMI credentials in Orion is required because the accounts will instantly get locked out and cause a LOT of errors after you change the password in Active Directory and the monitors keep trying to use the old password.)

• Stop all Orion services.
• Change the Orion service account password.
• Run the Configuration Wizard on all servers to update the credentials used for the service and for the connection to SQL Server.
• Restart all Orion services or servers.
• Pause monitoring of Windows servers where WMI credentials are used (stop the collection service or unmanage these nodes).
• Change the WMI credentials in Active Directory.
• Update the WMI credentials in Orion.
• Resume monitoring.

Notes / To-Do:

We will update this document ASAP with more details:

• Link to KB article with requirements for Orion service accounts.
• Link to KB article with instructions for how to create WMI monitoring accounts that do not require administrator privileges.

EDITS

2020/12/16 12:40 EST - Added to list of credential types in situation description.

2020/12/16 15:03 EST - Added link and note about support for running services as NETWORK SERVICE.

2020/12/16 20:54 EST - Added more information regarding Exchange mailbox accounts.