19 Replies Latest reply on Nov 16, 2016 4:44 AM by oiram

    Orion So Slow :(

    blueboy1878

      Hi

      I am wondering if someone can help me try to increase the performance of our Orion environment.

       

      Our Orion Server is:

      Physical Server

      Windows 2008 R2 Server Enterprise Edition

      2 * Intel XEON CPU E5-2670 V3 @ 2.30GHz Processors   

      64GB Ram

      HP SSD Discs - 185.96Gb in Size with 87.4GB free

      1 * 1GB Network Card

       

      Database Server

      Virtual Server

      8 vCPU

      24GB Ram

      Windows Server 2012 R2

      Microsoft SQL 2014

       

      We have the following Products:

      Orion Platform 2015.1.2

      SAM 6.2.1

      DPA 9.2.0

      QoE 2.0

      OFSMM 6.6

      IPAM 4.3

      NCM 7.4

      NPM 11.5.2

      NTA 4.1.2

      Storage 5.1.0

      UDT 3.2.2

      IVIM 2.1.0

      VNQM 4.2.2

       

      Netflow, Firewall Manager and Storage Manager all have their own dedicated servers (all Virtual)

      Firewall Manager

      Windows Server 2008 R2 Standard

      8vCPU

      8GB Ram

       

      Storage Manager

      Windows Server 2008 R2 Standard

      2 vCPU

      8GB RAM

       

      Netflow

      Windows Server 2012 R2

      4 vCPU

      16GB Ram

       

       

      I would really appreciate any pointers people can throw my way as its is unbearably slow, even rebooting the whole environment doesnt seem to improve it

        • Re: Orion So Slow :(
          lfransen

          I think the biggest area to look at is your SQL server. Solarwinds does not recommend using a virtual SQL server. I believe this is what they spec of the SQL:

          OS - Server 2012 R2 64bit

          SQL – 2008, 2012, 2014 : Standard version, latest SP applied (2012/2014 preferred)

          2 x 6 Core CPU or better

          32GB RAM

          2 x GbE NIC

          100 GB OS Storage (C: drive)

          50 SQL Application Storage (D: drive)

          250GB MS SQL Data file storage (E: drive)

          100GB MS SQL Log file storage (F: drive)

          50GB TempDB storage (G: drive)

          1 of 1 people found this helpful
            • Re: Orion So Slow :(
              mesverrum

              This^^

               

              Database is almost always the choke point when I deal with clients who have slow environments.  Is the SQL a dedicated box or does it run databases of other products as well?  The disks should be configured as lfransen describes and you need to confirm that the SQL files are actually on the appropriate drives (been to places with the right disk layout and all the important db stuff piled into C:)  Is the memory reservation set up correctly within SQL itself?  A DBA should be able to help you identify sources of excessive waits and locks here and confirm that appropriate maintenance tasks are configured within SQL to maintain the DB.  Check the built in SQL report of the top tables and see which ones are taking the most space.  I've been to a good number of places where the Traps and Syslogs tables are the biggest part of the db and there are no rules configured to even act on the data and they almost never use them.  If you do use them and it is crushing your NPM database maybe you need to move up to Kiwi or LEM as a dedicated system for that kind of thing.   NCM is another common culprit.  Are you retaining nightly config backups for the whole past year instead of only keeping the configs that changed.  Are you not archiving them for 6 months and does that provide you operational value or does it just bloat the db because and whoever set NCM up figured you had plenty of storage so why not?

               

              Since the db server is virtual we need to question your storage backend, how does its throughput look?  Does it have a flash tier?  Is your sql disk dumped into a pool with a hundred other random VM's and are the disk partitions that are supposed to be raid 10 in a correctly configured pool or are all pools built on 5/6/50/60 because that is "usually good enough" (the latter is the case for 95% of SAN's I've looked at).

               

              That's the places I would be starting my hunt if the performance is really awful.

               

              -Marc Netterfield

                  Loop1 Systems: SolarWinds Training and Professional Services

            • Re: Orion So Slow :(
              dmartzall

              Also, how many nodes are you monitoring?  I didn't see mention of an additional poller.

              • Re: Orion So Slow :(
                dmartzall

                The main server looks to be beefy if the CPU spec Google led me to was accurate you have 24 total cores?  It looks like you have 9 things running on this box and three additional servers each with a single purpose.  What I would look at is the "Total Job Weight" of your polling engines.  I have found that if you can keep that number under 3100, then it moves pretty quick.  It should look something like this:

                 

                Polling Rate 41% of its maximum rate.
                Routing Polling Rate0% of its maximum rate.
                Hardware Health Polling Rate32% of its maximum rate.
                UnDP Polling Rate0% of its maximum rate
                Wireless Polling Rate1% of its maximum rate.
                Fibre Channel Polling Rate0% of its maximum rate.
                F5 Polling Rate0% of its maximum rate.
                Wireless Heat Map Polling Rate0% of its maximum rate.
                Total Job Weight2635

                If the Polling rate is over 80%, I'd look to re-balance the load between your primary poller and any additional polling engines you may have.

                • Re: Orion So Slow :(
                  shaun_9999

                  We have the same problem here and run about 1,000 nodes on a similar spec system - save the SQL server is physical on fast fibre channel storage.and hardly any applications installed and it runs like a dog sometimes.

                  • Re: Orion So Slow :(
                    alexslv

                    The below steps is the first thing I always do for every instance of SolarWinds I have deployed. The level of improvements at GUI side (loading resources on page) is simply staggering!!! It just starts flying Not sure why SW Team would not make it as a default option out-of-the-box

                     

                    1. Stop All SolarWinds Services

                    2. Go to C:\Program Files (x86)\SolarWinds\Orion
                    3. Look for SolarWinds.BusinessLayerHost.config
                    4. Edit it
                    5. Look for <add key="ForcePluginsInSeparateProcess" value="False"/>
                    6. Change "False" to "True"
                    7. Restart All Services

                    Best luck... and fasten your belt! lol

                    Alex

                    1 of 1 people found this helpful
                      • Re: Orion So Slow :(
                        pparsaie

                        I think I see why you would do this just from the name of the key. What's the benefit and does this increase any CPU/MEM usage? Would you do this only on the 11.5.x versions or 12.x.x as well?

                        • Re: Orion So Slow :(
                          byrona

                          I am guessing this isn't a default because they typically tune the system for the lowest common denominator, not necessarily for best possible performance. 

                          • Re: Orion So Slow :(
                            oiram

                            I do the same, but bear in mind that in Core 2016+ there is small change. In these releases look at the Advanced Configuration options you can access through Web Console in order to set this option (and some other handy ones as well):

                             

                            /Orion/Admin/AdvancedConfiguration/Global.aspx

                            /Orion/Admin/AdvancedConfiguration/ServerSpecific.aspx

                             

                            Option you have mentioned - ForcePluginsInSeparateProcess - you can access on both of these pages, depends on what you need. I always set the global one to True.

                          • Re: Orion So Slow :(
                            byrona

                            As has been suggested it's most likely a database issue.  Try running the scripts below on your database and see if that helps any:

                             

                            Orion SQL Cleanup Scripts.zip

                            • Re: Orion So Slow :(
                              jkuvlesk
                              1 of 1 people found this helpful
                              • Re: Orion So Slow :(
                                dbascottclark

                                Ensure your DBA has enabled index maintenance processes on your Solarwinds database.  These are crucial to the performance of the application.  Our findings have shown that not performing an index maintenance routine daily had a huge impact on the performance of the SolarWinds UI.

                                  • Re: Orion So Slow :(
                                    alexslv

                                    Would you please be able to outline what exactly needs to be done with those indexes - just for the sake of those who do not have DBAs on site. Thanks a million

                                      • Re: Orion So Slow :(
                                        mbird

                                        I'm interested as well.

                                          • Re: Orion So Slow :(
                                            dbascottclark

                                            There are a multitude of approaches and strategies which need to be evaluated based upon each organization use patterns for Solarwinds.  For our case we leverage an internally developed script to maintain the health of the indexes on all tables within the NPM database.  If you are looking for a solution which takes most of the guess work out of configuring something for your organization based upon best practices the following scripts from Ola Hallengren https://ola.hallengren.com/sql-server-index-and-statistics-maintenance.html is a really good starting point.  They do have a bit of a learning curve for setting them up the first time and ensuring your choosing the correct index maintenance operations.  In addition, your strategy may be different than ours if you aren't running SQL Server Enterprise Edition (2012, 2014) which allow for On-line re-indexing operations which reduces blocking caused by the re-build operation.

                                             

                                            Give Ola's site a read as for the Non-DBA Database Admin Ola does a great job of making your job easier.

                                            2 of 2 people found this helpful
                                      • Re: Orion So Slow :(
                                        grantallenby

                                        Something to be very aware of apart from hardware is limitations.

                                         

                                        We have very high spec'd kit and see some very slow speeds, this is due to the nature of our environment and limitations on nodes. We have 7 polling engines and limit all nodes down to area and then limit accounts based off of this. we also use page limitations to name a few. This unfortunately dramatically slows the platform down.

                                         

                                        I have tested it by a simple limitation of 40 nodes and it fly's. Apply the limitation to much larger environments and add more complex limitations and it noticeably reduces how fast nodes will load.

                                         

                                        A carefully planned limitation usage shouldn't be over looked.

                                        • Re: Orion So Slow :(
                                          drdoomprophet

                                          The only time I have lag/slowness issues is when I try to run reports. But I figure that's due to having to parse through all the data.

                                          • Re: Orion So Slow :(
                                            dragoon231

                                            The database is always the first thing I look at, the other thing that may go unnoticed as well is the webconsole.  I think a single server can handle 25 open web consoles at a time (I read that somewhere i think), every time a user navigates through pages, those are queries happening for the pretty information you see (charts, averages, etc).  If it is more, you will need to get an additional license for a server to  handle more web consoles.  One tool I like to use is hubble (on your Orion server) for reviewing query times in the webconsole as you navigate through pages to figure out what is happening.