cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

When you have to change all passwords for the Orion monitoring platform

File under #2020:
Situation


You are required to change all Orion-related passwords in response to a security breach or some other event. The affected credentials may include the following list, and possibly more:

 

  • A domain-based service account used for the Orion platform's Windows services*
  • A domain-based account used to access the SQL database (or a SQL-based login)
  • Domain-based accounts used for monitoring Windows servers and applications with WMI
  • A domain-based account with an Exchange Server mailbox for sending alerts
  • SNMP community strings and/or SNMPv3 credentials
  • Application credentials
  • Hypervisor credentials in NPM, VMAN, or SAM
  • SSH credentials in NCM


This write-up will focus on the first four types of accounts, which should all be domain-based accounts in Active Directory, unless you are running Orion services as the default NETWORK SERVICE security principle. 

* You may find that your Orion services are running as NETWORK SERVICE instead of a service account. Per SolarWinds documentation, this is a supported configuration.

 

Solution

 

Let's skip to the spoiler: in this kind of scenario, the least impactful and most secure approach is to create new domain-based accounts instead of changing passwords.

 

Step 1: Orion Service Account

  • Create a new service account for Orion in Active Directory and grant it the appropriate permissions on your Orion servers.(Or let them just run as NETWORK SERVICE, if that fits your environment's requirements.)
  • Coordinate with your DBAs to grant this new service account the required permissions on your Orion database[s].
  • Run the configuration wizard on each of your Orion servers, providing the new credentials for the services and for the connection to the SQL database[s].
  • Once you have confirmed that the Orion services are working properly, the original service account should be disabled.

 

Step 2: WMI Monitoring Accounts

  • In each Active Directory forest that you monitor, it is best to create at least 3 accounts for monitoring Windows servers:
    1. One for monitoring your domain controllers (regardless of which template you use).
    2. One for monitoring your Exchange servers with AppInsight for Exchange.
    3. One for monitoring all other member servers.
    4. Also: use a dedicated account for monitoring your SharePoint farms (if you have SharePoint) and a dedicated account for the AppInsight for SQL Server template, as both of these require special permissions for the service account.
  • Add these new WMI account credentials to Orion.
  • Assign these new WMI credentials to the relevant nodes.
  • Once you have confirmed that nothing else is using your old WMI monitoring accounts, they should be disabled and removed from any privileged groups.

 

Step 3: Mailbox Accounts

 

How are your alert emails and job notification emails sent from Orion? If you have created an Exchange mailbox for your original service account to send alerts, here is another best practice to consider: use a separate account with its own mailbox. When using the same account for email and for the application itself, you increase your risk of an accidental account lockout bringing down the whole application. This has potential to happen any time the SMTP settings are changed for the Orion platform, and any time an NCM job's email notifications are set.
Does this one require a brand new account and brand new mailbox? I'll leave that up to you and your security team. A password reset may be deemed sufficient. If new accounts are mandated across the board, then be prepared to help recreate any inbox rules that your alert recipients have configured in their mail clients.

 

Step 4: Final Validation and Cleanup

 

These final steps may vary depending on how your environment is configured and what your policies around account lifecycle require.

 

  • Does anything else utilize your WMI monitoring accounts for similar purposes? It would be best to not share these accounts, but use credentials specific to each platform. Work with your Active Directory team and/or Security team to audit domain controller logs and make sure that these accounts are not still being used by anything else.
  • What is your account lifecycle policy? You may be required to disable but keep old accounts for security purposes. Check any safeguards that are in place to make sure that these cannot be re-enabled and re-used, or delete these accounts after confirming that they are no longer in use anywhere.


Q: Why shouldn't we just change the passwords?


This can certainly be done, but there is actually more work involved, more potential disruption, and ir provides less guarantee of mitigation if any of your accounts were ever compromised. At a very high level, the process for this approach would include the following steps.

(Temporarily stopping the use of WMI credentials in Orion is required because the accounts will instantly get locked out and cause a LOT of errors after you change the password in Active Directory and the monitors keep trying to use the old password.)

 

  • Stop all Orion services.
  • Change the Orion service account password.
  • Run the Configuration Wizard on all servers to update the credentials used for the service and for the connection to SQL Server.
  • Restart all Orion services or servers.
  • Pause monitoring of Windows servers where WMI credentials are used (stop the collection service or unmanage these nodes).
  • Change the WMI credentials in Active Directory.
  • Update the WMI credentials in Orion.
  • Resume monitoring.

 

Notes / To-Do:

We will update this document ASAP with more details:

  • Link to KB article with requirements for Orion service accounts.
  • Link to KB article with instructions for how to create WMI monitoring accounts that do not require administrator privileges.

 

EDITS

2020/12/16 12:40 EST - Added to list of credential types in situation description.

2020/12/16 15:03 EST - Added link and note about support for running services as NETWORK SERVICE.

2020/12/16 20:54 EST - Added more information regarding Exchange mailbox accounts.

Tags (1)
8 Replies
Level 12

Any update on when we get information on this one:  "WMI monitoring accounts that do not require administrator privileges"

This would be especially helpful information to document better.  I'm surprised @aLTeReGo  and the SAM team haven't provided anything about this and for security best practices for implementing: SAM templates that leverage powershell, AppInsight for Active Directory and other templates that could be used to monitor a Windows Active Directory environment

0 Kudos
MVP
MVP

You write a mighty fine post! And the organized edit notes at the bottom? :chefkiss: perfection!

0 Kudos
Level 8

Any guidance for mass updating the creds for each node or is it node by node?

Thanks for the guide btw...

0 Kudos

Which creds, specifically? Service accounts for Orion nodes, or the WMI creds that you monitor Windows servers with?

Happy to help.
0 Kudos

In this case WMI.

0 Kudos

I have not yet had a chance to automate changing of the WMI credentials on nodes, but that can certainly be done. (Thankfully I'm not in a place that I must change them, but am going to do so anyway.)

The best starting point would be to create a custom Orion report with the following SQL query and review it. This will show you all of your nodes grouped by which WMI credentials they use. Of course, it shows the saved name of the credential, and not the credential itself.

 

select 
  Credential.Name, 
  nodes.Caption, 
  nodes.NodeID, 
  nodesettings.SettingName, 
  NodeSettings.SettingValue 
from 
  Nodes 
  join NodeSettings on nodes.NodeID = Nodesettings.NodeID 
  join Credential on nodesettings.SettingValue = Credential.ID 
where 
  nodes.ObjectSubType = 'WMI' 
  and nodesettings.SettingName = 'WMICredential' 
order by 
  Credential.Name, 
  nodes.Caption, 
  nodesettings.SettingValue asc

 

This will help you grasp the scope of change that needs to be made, and similar SQL can be used for the automation part. 

When you refer to the 'automation' part, do you mean SQL can be crafted to change the assigned Windows Credential for each node via this method?

This would really help me personally, since I'm at a place where I've got a broken install - the website throws errors when I attempt to edit any WMI node, regardless of credential used. If I could just do this via the DB, I'd have one less problem to deal with. I'm waiting on support, but it's been three days and I haven't heard squat from them.

Thanks for any info.

0 Kudos

Sorry all of the formatting got tossed somehow. I'll fix it try to revert asap in the morning!
0 Kudos