Shutdown & Restart Order

In the event of maintenance and documenting I'm looking for a detailed process of how you should shutdown the SolarWinds Orion environment.

1) Switch off all High Availability services prior to executing this.
2) SQL
3) Primary Poller
4) Additional Poller and Web Servers
5) HA's

Would this be the correct order in which to shutdown.

Also I would presume that the power on sequence should be;

1) SQL
2) Primary Poller
3) Additional Poller and Web Servers
4) HA's
5) Switch on High Availability services

Find more posts tagged with

power on

order

shutdown

sequence

Accepted answers

sum_giais

I'm not aware of any official documentation that speaks to this however I do know that there's some documentation that speaks to pieces of it.

You're on the right track though.

Shutdown: (Note: Startup is exact opposite)

Pause all Orion alerts if desired
Disable HA (High Availability)
Stop services and/or shutdown Additional Web Servers
1. Open Orion Services Manager and "Shutdown Everything"
Stop services and/or shutdown HA Standby servers
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown any tertiary servers (wpm locations etc., depending.. these could be omitted)
Stop services and/or shutdown Additional Polling Engines
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown Main Polling Engine
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown SQL Database Server

Startup:

Start services and/or power-on the SQL Database Server (give this a few ~2m depending to startup)
Start services and/or power-on Main Polling Engine
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power on Additional Polling Engines
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power-on any tertiary servers (wpm locations etc.)
Start services and/or power-on HA Standby Engines
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power-on Additional Web Servers
1. Open Orion Services Manager and "Start Everything"
Enable HA (High Availability)
Resume all Orion alerts if desired

This has always been my order of operations for shutdown and startup within any Orion environment and have never had any issues doing it this way. Much of SolarWinds own documentation supports this but doesn't directly go into such detail sadly.

Additionally the whole process can be scripted from start to finish if you want to go that far.

Edit (01/29/2022):

I added some additional clarity and also detailed out the more of the process. I'm pretty meticulous when it comes to the startup because a proper startup procedure can ensure a smooth, clean running Orion environment. I've never seen any documentation that references to the dependencies and in certain cases depending on WHY you're shutting down services you may not want to actually go that far as to stop dependencies. For patching of the servers it's good practice to stop EVERYTHING though. But technically the TFTP server and SFTP/SCP server services are not "Orion" services (at least that's what I was told by SolarWinds Support) - feel free to correct me on that.

All comments

sum_giais

I'm not aware of any official documentation that speaks to this however I do know that there's some documentation that speaks to pieces of it.

You're on the right track though.

Shutdown: (Note: Startup is exact opposite)

Pause all Orion alerts if desired
Disable HA (High Availability)
Stop services and/or shutdown Additional Web Servers
1. Open Orion Services Manager and "Shutdown Everything"
Stop services and/or shutdown HA Standby servers
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown any tertiary servers (wpm locations etc., depending.. these could be omitted)
Stop services and/or shutdown Additional Polling Engines
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown Main Polling Engine
1. Open Orion Services Manager and "Shutdown Everything"
2. Manually stop the dependencies with Orion Service Manager
3. Open Windows Services Console to shutdown:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
Stop services and/or shutdown SQL Database Server

Startup:

Start services and/or power-on the SQL Database Server (give this a few ~2m depending to startup)
Start services and/or power-on Main Polling Engine
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power on Additional Polling Engines
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power-on any tertiary servers (wpm locations etc.)
Start services and/or power-on HA Standby Engines
1. Open Windows Services Console to startup:
  1. SolarWinds High Availability
  2. SolarWinds SFTP/SCP Server / TFTP Server etc.
2. Manually start the dependencies with Orion Service Manager (wait ~10-15s before starting all services)
3. Open Orion Services Manager and "Start Everything"
Start services and/or power-on Additional Web Servers
1. Open Orion Services Manager and "Start Everything"
Enable HA (High Availability)
Resume all Orion alerts if desired

Additionally the whole process can be scripted from start to finish if you want to go that far.

Edit (01/29/2022):

bobmarley

That is the same order we use, has always worked fine. Only time we ever have issues is if the SQL server gets pulled out and then it sometimes would not process the queues afterwards and needed a manual process to fix those.

donrobert5

https://support.solarwinds.com/SuccessCenter/s/article/Orion-server-reboot-Best-practices?language=en_US

However, this does not include the HA. I would say yours is spot on.

SW_United

This is exactly what I was looking for @sum_giais, excellent thanks so much for your help.

As for the scripting, that would be awesome if you could assist with this; I've already done some research into this and I've got a PS script for VMware CLI, what do you think?

$vm_list = Get-Content "vmlist.txt"
$vcenter = ""

Connect-VIServer $vcenter 

foreach ($vm in $vm_list){

Shutdown-VMGuest -VM $vm_list -Confirm:$False
}

Disconnect-VIServer -Confirm:$false

sum_giais

I'm not very familiar with the VMware CLI in powershell but are you looking to script out simply shutting down the servers and then starting them back up?

SW_United

Essentially, yes. I was trying to automate this task and other admin tasks as much as possible and me be more of observer of the script/s. [tag:LazyAdmin]

Holcomb

If the Terracotta servers were not configured as restartable, or the servers' data was wiped before restarting the servers, then clients will fail to reconnect to the server and require restart anyway.

MyKohlsCharge