7 Replies Latest reply on Jun 4, 2014 1:12 PM by curtisi

    Slow reports

    seffland

      We need to generate some reports for an audit coming up and the report builder has been PAINFULLY slow. We have about 600,000 entries going into each report and they take anywhere from 30 min to 1.5 hours to create. The VM that the LEM is running on is certainly up to spec and we're only monitoring 50 nodes. Any suggestions would be great!

       

      Thanks!

        • Re: Slow reports
          curtisi

          In regards to reports, can you please check the following?

           

           

          1.  For LEM 5.6, have you deployed Hotfix 1?

            - If you SSH into your LEM, go to the MANAGER menu and then run VIEWSYSINFO, the TriGeo manager build should be "hotfix1"

          2.  For Windows Vista and above, was the Reports console installed as an Administrator?

            - If you're not sure, can you do the following, please?

            1. Navigate to the Reports console Programs folder, ie C:\Program Files (x86)\SolarWinds Log and Event Manager Reports

            2. Copy the CustomReports folder to a safe location

            3. In the "Programs and Features" or "Add and Remove Programs" Control Panel, un-install the Solarwinds Reports Console

            4. Un-install the Crystal Reports 11 Runtime

            5. Delete the Reports program folder

            6. While you're in the Program Files folder, see if you have a Contego Console folder.  Rename it to something like Contego Console.old (only applies if you ran older versions of LEM)

            7. Navigate to C:\Users\(YOUR USER)\AppData\Roaming

            8. Rename the sim.console folder to something like sim.console.old (Only applies if you've ever used the LEM AIR Console)

            9. Download the installers for the Reports Console and the Crystal Reports runtime from the Solarwinds Customer Portal

            10. Install the Crystal Reports runtime as an Administrator (right-click, Run As...)

            11. Install the Solarwinds Reports Console as an Administrator (right-click, Run As...)

            12. Copy the CustomReports folder back to the program directory

          3.  For Windows Vista and above, are you running Reports as an Administrator?

          4.  If you run the Report for a shorter period of time, like the last hour or last day, is it still slow?

          5.  If you install the Reports and runtime on a different machine, is it still slow?

            - You may need to SSH into your LEM, go to the SERVICES menu and run UNRESTRICTREPORTS to allow another machine to access the database

          6.  You may want to tweak Java to give more memory to the Reports console

            - Close Reports

            - Navigate to C:\Program Files (x86)\Common Files\Business Objects\3.0\java

            - Open CRConfig.xml

            - Locate the following:

            <IORFileLocation>${TEMP}</IORFileLocation>

            <JavaServerTimeout>1800</JavaServerTimeout>

            <JVMMaxHeap>64000000</JVMMaxHeap>

            <JVMMinHeap>32000000</JVMMinHeap>

            <NumberOfThreads>100</NumberOfThreads>

            </DataDriverCommon>

            <JDBC>

            - Change the value in the JVMMaxHeap tag to either 256000000 or 1024000000

            - Change the value in the JVMMinHeap tag to 64000000

            - Save

            - Reopen Reports and try again

          7.  Are all reports slow, or just specific reports?

            • Re: Slow reports
              seffland
              1. We are on LEM 5.7
              2. Yes
              3. Yes Reports always launches as admin (changed in properties)
              4. No it only seems to be slow when run over 3+days (We need 1 week at a time)
              5. When I went to install reports I get this error
                1. Capture.PNG
                2. It says something very briefly about having to obtain a license
              6. I did this to no luck.
              7. And some seem to be fast, but the "Network Events: Suspicious Behavior" (Master) seems to be the worst out of the lot. It will actually freeze up reports when run for the 1 week time period.

              Thanks so much for your help!

                • Re: Slow reports
                  curtisi

                  RE: 5 - That sounds like you need to reinstall Crystal Reports Runtime as an Administrator!

                   

                  There may be no better way to make things faster than to shrink the time-frames on those reports and run them for 3 or 4 day periods.

                    • Re: Slow reports
                      seffland

                      Uninstalled and re-installed as admin with no luck.

                       

                      And It just seems crazy that you can't generate a report with a time-frame longer than 3 days...

                        • Re: Slow reports
                          curtisi

                          Ready to dive into the LEM techronomicon?  Here we go...

                           

                          The LEM generates new partitions as event data comes in, and drops old partitions as the disk fills up. The frequency of this creation is based partially on size and partially on event count.  If you want to see all the partitions, you can run the Database Maintenance Report, and it'll list all of them.

                           

                          Now, if you were to root into the LEM and look at the alert DB, you'd see that some number of the newest partitions are big, like a gigabyte or so.  These are the "Warm" partitions.  The rest of them are a lot smaller, usually in the neighborhood of 200 to 500 MB.  The reason for this is the LEM has compressed the partition a bunch and removed the "warm" files to increase the amount of data you can store on the LEM.

                           

                          The number of partitions the LEM keeps "warm" is based in part on the memory you have assigned to the LEM and on what we've found is most optimal in testing, but we strive for a week of data warm at any given time.  If you're searching for a date range in the warm partitions, searches should be pretty quick: all the LEM has to do is execute the query.

                           

                          But let's say that you're going way back in time, and searching.  Now the LEM has to find the partition(s) with the data you want in them, load them into memory, decompress them, run the query, recompress them and provide you the results.  This takes extra time and resources on the box, so searches in the past will always be slower but it means you can retain more data on a smaller disk.

                          1 of 1 people found this helpful
                            • Re: Slow reports
                              seffland

                              Thank you for the explination!

                               

                              Is it possible that the sheer number of alerts we are receiving every min could cause this "warm" period to shrink? And if so how would we go about removing alerts we don't need?

                               

                              Thanks again!

                                • Re: Slow reports
                                  curtisi

                                  Yes, because memory is a factor, and the LEM will prioritize processing new events and keeping rules firing over the database partitions being warm.  Our assumption is you want your real-time alerts more than you want to search for things.

                                   

                                  As far as "removing" alerts, the LEM is passive.  It's like a mailbox: it doesn't go out and get mail, it just takes whatever you get delivered.  If your mailbox is full of chain-letters and magazines, you should consider reducing the number of pen-pals and subscriptions you have.  If you've got a lot of alerts that you don't want or need, kill them at the source!

                                   

                                  Ironically, I just posted a thing related to that... http://thwack.solarwinds.com/docs/DOC-174865