This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

My Orion Website & Server is very slow / CPU & Memory Spike /Polling gaps what should i check quickly ?

This article provides quick information about  your current environment and health check , Further it  will help you address the most common reasons of  performance issues on to your server without sending the diagnostics to SolarWinds support  .

In this article you can also Audit your own environment quickly if its been setup as recommended by the Solarwinds MINIMUM requirements or according to the settings eliminating bottlenecks creating performance issues within the set environment .

This article also help to save time to upload the diagnostics for Support where you have air gap between the server and you can check the basic health check on the actual server itself .

There are lots of other benefits to check the environment health within the internal due to security procedure not allowing uploading the diagnostics for Solarwinds support .

AAEAAQAAAAAAAAS2AAAAJGU1ZjVjN2NkLTIwYjgtNGI4Zi1iMmZjLWUzOWM4M2MzYTg3OQ.jpg

Your check list

 Server Hardware

 Total Elements (Nodes /Interfaces/ Volumes)  been polled per server

 Check free disk space on the Orion Server and SQL server

 Check Your Server Polling  Rate

 SQL Server / Orion DB Size / Settings / Options

 Check SQL Server Disk Performance

 Orion Antivirus directory exclusion

 Webpages Customization

Lets Go!

Collect System diagnostics as below.

Navigate to Start -> SolarWinds Orion -> Documentation and Support

Launch the gray icon for Orion Diagnostics.> Click "Start"

This program will generate a .zip file as output.

( Unzip in a folder ) Right Click >  Select Extract Here .

Server Hardware

Lets check your System Hardware first if this even near to the Solarwinds MINIM recommended.

Go to the SystemInformation folder > Open the SystemInfo.txt file

pastedImage_2.png

You will be able to find the System hardware specification  below is an example where system is only assigned with 2 Physical CPU Sockets /

below is an example where the system is only assigned two CPU PHYSICAL SOCKETS which is below Solarwinds MINIMUM recommendation .

You must have to have MINIMUM 4 PHYSICAL CPU SOCKETS here .

System Type:               x64-based PC

Processor(s):              2 Processor(s) Installed.

                           [01]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~1600 Mhz

                           [02]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~1400 Mhz

                          Total Physical Memory:     49.082 MB

                          Available Physical Memory: 39.408 MB

                          Virtual Memory: Max Size:  56.250 MB

                          Virtual Memory: Available: 45.376 MB

                          Virtual Memory: In Use:    10.874 MB

Now open the SysInfo.csv file and check the further current CPU load on the System and CPU GHz level .

pastedImage_1.png

Below in an example where the CPU load in around 70% on the current system due to two main reasons .

Parameter

Value

OSVersion

Windows Server 2012 R2 (Microsoft Windows NT 6.2.9200.0)

CPUInformation

Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz

CurrentCPUUssage

70 %

TotalPhysicalMemmory

49152 MB

FreePhysicalMemmory

39802 MB

FreeVirtualMemmory

45843 MB

FreeSpaceInPagingFiles

7109 MB

CurrentTimeZone

xxxx Standard Time (UTC+01:00:00)

Low Physical Sockets assigned

Low CPU power less then 3.0 GHz

You should be able to see MINIMUM 4 Physical Processors  Sockets as below .

Strongly recommend :  NOT to use lower then 3.0 GHz processor you will never get the performance what you are looking for even the Host and Guest wont show the CPU is busy .

Most likely you will see CPU spikes / Orion Services consuming High CPU and Memory . Once you will move the same VM to higher then 3.0 GHz process all the above symptoms will be resolved.

With lower then 3.0 GHz processor there might be other issues such as SQL Server TCP connections TimeOut Errors and High amount of data  stored under MSMQ on the system .

pastedImage_0.png

Make sure you have MINIMUM 3. Ghz host with Hyperthreading Active it will improve the Guest performance significantly and you will have full performance out of Solarwinds application

pastedImage_2.png

This is how you setup your VM in ESX

pastedImage_0.png

Here is an Example when you assign the numbers of CPU SOCKETS to the VM

pastedImage_0.png

System Model:              VMware Virtual Platform

System Type:               x64-based PC

 Processor(s):              4 Processor(s) Installed.

                           [01]: Intel64 Family 6 Model 15 Stepping 1 GenuineIntel ~3493 Mhz

                           [02]: Intel64 Family 6 Model 15 Stepping 1 GenuineIntel ~3493 Mhz

                           [03]: Intel64 Family 6 Model 15 Stepping 1 GenuineIntel ~3493 Mhz

                           [04]: Intel64 Family 6 Model 15 Stepping 1 GenuineIntel ~3493 Mhz

BIOS Version:              Phoenix Technologies LTD 6.00, 4/14/2014

If you are on HyperV can adjust VM Sockets under NUMA  for more details please see below posts

Hyper-V Design for NUMA Architecture and Alignment - | Exit | the | Fast | Lane |

https://www.starwindsoftware.com/blog/a-closer-look-at-numa-spanning-and-virtual-numa-settings

image

pastedImage_1.png

Further check how much memory is assigned and available for the system and check the TaskManager which application is consuming high memory .

In above case the System Hardware is not even near to the recommended SolarWinds production deployment therefor the CPU load will remain high therefor System resources.

The following table lists minimum hardware requirements and recommendations for your SolarWinds Orion server.

Installing multiple SolarWinds Orion Platform products on the same computer may change the requirements.

Hardware requirements are listed by SolarWinds NPM license level.

NPM hardware requirements

These minimum requirements are for the Orion Platform. Products that run on the Orion Platform may have different requirements, such as different OS or memory requirements.

Consult your product-specific documentation for the exact requirements.

Hardware

SL100, SL250, SL500

SL2000

SLX

CPU speed

Quad core processor, 2.5 GHz or better

Quad core processor, 2.5 GHz or better

Quad core processor, 3.0 GHz or better

For more details see below guide

NPM 12.0 system requirements - SolarWinds Worldwide, LLC. Help and Support

Check free disk space on the Orion Sever and SQL Server

Make sure you have Good free space available on the Orion Server disks C Drive and installed Directory .

Make sure you have Good free space available on the SQL Server where the actual DB is stored.

Total Elements (Nodes /Interfaces/ Volumes)  been polled per server

Go to folder "DB" > Open file "AllEngines.csv"

pastedImage_4.png

Check how many Elements you are polling per server

EngineID

Elements

Nodes

Interfaces

Volumes

1

15828

934

6823

1071

2

16084

202

1305

77

With only SolarWinds SLX license you can montior up to 12000 Elements and beyond this you will need an Additional Polling Engine to monitor.

More Details see the Server Sizing guide .

Server Sizing recommendations

Use additional polling engines for 12,000 or more monitored elements

If you plan to monitor 12,000 or more elements, SolarWinds recommends that you install additional polling engines on separate servers to help distribute the work load.

I would also strongly advise you to check the blog post for any other questions if you are polling beyond 12000 Elements with single SLX Server.

Boost your server polling capacity with Stackable Poller license

Multi-module system guidelines

Check your Server Polling Rate

Go to Settings > Polling Engines .

Check if any of the Polling Rate is increased ?

Make sure none of the Polling Rate exceeded above 100%

POLLING COMPLETION100
ELEMENTS225
NETWORK NODE ELEMENTS18
VOLUME ELEMENTS50
INTERFACE ELEMENTS157
POLLING RATE2% of its maximum rate.
ROUTING POLLING RATE0% of its maximum rate.
HARDWARE HEALTH POLLING RATE0% of its maximum rate.
VIM.VMWARE.POLLING2
F5 POLLING RATE0% of its maximum rate.
WIRELESS HEAT MAP POLLING RATE0% of its maximum rate.
WIRELESS POLLING RATE0% of its maximum rate.
UNDP POLLING RATE0% of its maximum rate.
SAM APPLICATION POLLING RATE                       170% of its maximum rate.             

If you have any polling rate increased above the 100% you will notice high CPU / Memory Utilization on the System which could effect the System and application Performance .

Orion DB Size and settings

Go to the DBInfo Folder > Open DatabaseInfo.csv file

pastedImage_0.png

Check the Database Recovery Mode

Check the Total Database Size

Default DB Recovery should be SIMPLE (Strongly recommended)

name

db_size

status

SolarWindsOrion

889274.25 MB

Recovery=FULL

  • I have a very large Orion database. What should I check?
  • I have database performance issues. How can i improve my database performance?
  • Why is my Orion database growing so quickly?
  • Why does my Orion performance decrease each day?
  • Why is my Orion Web Console loading data at a slow rate?

For more details please see the post below and follow all the steps one by one to check your Orion Database Health and settings.

This guide will help you address the most common questions and issues related to the Orion database performance check and configuration without using the SolarWinds Database Administrator (DBA).

Quick Orion database health check guide

Check SQL Server Disk Performance

Orion Antivirus directory exclusion for NPM

Web pages recommended settings

Still have any question / need assistance ?

main.jpg

Please feel free to submit a new support ticket in relation to your question/error. Our support lines are available 24/7.

http://www.solarwinds.com/support/ticket

You can also contact the support by 24/7  phone support .

http://www.solarwinds.com/company/contact.aspx

Parents
  • Just resolved a case :

    Customer issue:

    CPU increases gradually and is resolved after a restart.

    Customer environment

    Currently we have high CPU utilization.We have 16 vCPUs (E5649 2.53 GHz)

    System Manufacturer:       VMware, Inc.

    System Model:              VMware Virtual Platform

    System Type:               x64-based PC

    Processor(s):              2 Processor(s) Installed.
                               [01]: Intel64 Family 6 Model 37 Stepping 1 GenuineIntel ~2533 Mhz
                               [02]: Intel64 Family 6 Model 37 Stepping 1 GenuineIntel ~2533 Mhz
    BIOS Version:              Phoenix Technologies LTD 6.00, 9/21/2015

    Windows Directory:         C:\Windows

    System Directory:          C:\Windows\system32

    Boot Device:               \Device\HarddiskVolume1

    System Locale:             en-us;English (United States)

    Parameter Value
    OSVersionWindows Server 2016 (Microsoft Windows NT 6.2.9200.0)
    CPUInformationIntel(R) Xeon(R) CPU           E5649  @ 2.53GHz
    CurrentCPUUssage93%

    pastedImage_0.png

    Resolved:

    Adding 2 more CPU SOCKETS to the VM and moved the VM to 3.4 Ghz  ESX host -

    Issue resolved and system is back on stable condition verified

Reply
  • Just resolved a case :

    Customer issue:

    CPU increases gradually and is resolved after a restart.

    Customer environment

    Currently we have high CPU utilization.We have 16 vCPUs (E5649 2.53 GHz)

    System Manufacturer:       VMware, Inc.

    System Model:              VMware Virtual Platform

    System Type:               x64-based PC

    Processor(s):              2 Processor(s) Installed.
                               [01]: Intel64 Family 6 Model 37 Stepping 1 GenuineIntel ~2533 Mhz
                               [02]: Intel64 Family 6 Model 37 Stepping 1 GenuineIntel ~2533 Mhz
    BIOS Version:              Phoenix Technologies LTD 6.00, 9/21/2015

    Windows Directory:         C:\Windows

    System Directory:          C:\Windows\system32

    Boot Device:               \Device\HarddiskVolume1

    System Locale:             en-us;English (United States)

    Parameter Value
    OSVersionWindows Server 2016 (Microsoft Windows NT 6.2.9200.0)
    CPUInformationIntel(R) Xeon(R) CPU           E5649  @ 2.53GHz
    CurrentCPUUssage93%

    pastedImage_0.png

    Resolved:

    Adding 2 more CPU SOCKETS to the VM and moved the VM to 3.4 Ghz  ESX host -

    Issue resolved and system is back on stable condition verified

Children
No Data