9 Replies Latest reply on Jun 29, 2016 7:49 AM by tomas.kuty

    VMAN On-Demand Reporting (and VMAN Reports in General)

    xtraspecialj

      Specifically, the OverSized VM's by Specific Cluster On-Demand report is what I'm asking about here.  One general problems with the report is that it isn't displaying a single cluster despite it being selected when we run it.  Instead it is showing all VM's in all clusters. 

       

      But, my question is about the "Recommendation" CPU and Memory columns that show up on the report.  Where does it get the "Recommendation" values from?  Do they come from the VM CPU/Mem OverAllocated out-of-the-box VMAN alerts' recommendations?  If so, how is that configured in the report to pull that data?  I can't really find any way to configure that report and none of the regular Reports seem to have the recommendation data, but  VMAN reports in general are about the most confusing dang thing I've ever come across and I could be missing it. 

       

      If anyone can shed some lights on these specific problems or just give me any kind of tips that would help me understand VMAN reports in general that would be greatly appreciated, because the Admin guide is DEFINITELY not any help.  It just tells you the steps on how to do things in the least words possible and doesn't tell you why or how any of the things work....

        • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
          xtraspecialj

          I know the VMAN forums aren't quite as busy as some of the others since there are a ton of unanswered questions on here I've noticed, but I'm hoping somebody can help me here.  Anyone?

            • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
              Milan_Volejnik

              Hi, sorry for late answer.

              1) if you are getting all VMs from all clusters then please contact SolarWinds support.

              2) you are right, the "Oversized VMs for a specific Cluster" report is based on combination of "VM CPU Overallocated" and "VM Memory Overallocated" alerts. However the definition is hardcoded in the report. So if you modify the alerts (either "Search Query" or "Notification value"), the On-Demand report will not be updated.

               

              And definition of the 'VM Memory Overallocation' alert:

              Search Query:

              vm.memory:[2048 TO *] AND vm.powerstate:poweredOn AND vm.consumedPercentMemload.week:[0 TO 30]
              

              - max. limit (30) of vm.consumedPercentMemload.week is customizable in the report

              - Search Query in the report contains also condition to select only VMs belonging to the selected cluster

               

              Notification:

              if(floor(/virtualMachine/consumedMemoryUtilization/week div 1024 * 1.5) < 256) 
              then concat('Reduce ', /virtualMachine/memory, ' MB to 256 MB') 
              else concat('Reduce ', /virtualMachine/memory, ' MB to ', floor(/virtualMachine/consumedMemoryUtilization/week div 1024 * 1.5), ' MB')
              

               

              The 'VM CPU Overallocation' alert:

              Search Query:

              vm.cpuCount:[2 TO *] AND vm.powerstate:poweredOn AND vm.cpuloadSumpeak.week:[0 TO 30]
              

               

              Notification:

              concat('Reduce ', /virtualMachine/cpuCount, ' to ', 
              (if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 80) then 1
              else if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 160) then 2
              else if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 320) then 4 else 8))
              

               

              I hope it will help you understand how the report works.

                • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                  xtraspecialj

                  Thanks for the response.  I ended up putting a ticket in because it is giving us VM's in clusters different than what we choose.  For example, lets say we have clusters A, B, and C in VMAN.  If we choose Cluster A for the report, we may get VM's from Cluster C.  If we choose Cluster C it might be empty and if we choose Cluster B it might be either empty, or from Cluster A.  It is really weird...

                   

                  I did find the reports in /data/var/hyper9/data/content/reports and was able to discvoer the hard-coded nature of them by reading the python script that generates the report (it's called OversizedVMsReport.py.)  I took this part out of it:

                  vm.origin:vmware AND vm.powerstate:poweredOn AND vm.cluster.name:%s AND ((vm.cpuloadSumpeak.week:[0 TO %s] AND vm.cpuCount:[2 TO *]) OR (vm.memloadPeak.week:[0 TO %s] AND vm.memory:[2048 TO *]))
                  

                   

                  and replaced the "%s" placeholders with my cluster name and thresholds, pasted it into the Search of VMAN and it returned the correct results, as expected.  So it seems to definitely be something weird where when we are choosing the cluster something in the web code is sending the wrong cluster name to this variable.  We have two VMAN environment monitoring two separate vCenter environments and both VMAN's return the wrong cluster no mater which cluster we choose.  What the heck??

                  1 of 1 people found this helpful
                    • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                      chrispaap

                      J

                      Send me your ticket number and I will track on our side.

                      • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                        Milan_Volejnik

                        It's really weird. Just some ideas come to my mind:

                        - are the cluster names unique and distinct?

                        - do the cluster names contain special characters (i.e. apostrophes)?

                        - is cluster A name a part of cluster C name?

                          • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                            xtraspecialj

                            - are the cluster names unique and distinct? Unique and distinct, yes.

                            - do the cluster names contain special characters (i.e. apostrophes)?  Dashes are the only special characters

                            - is cluster A name a part of cluster C name?  They do share some commonalities, see below:

                             

                            BOS - PROD - App

                            NC - Citrix - App

                            NC - Prod - App

                            etc...

                             

                            I would say this theory could be it, however If I choose Boston in the report it comes back with NC - Citrix - App vm data, and those two have nothing in common except for the App at the end...  Also, our other VMAN environment that looks at a completely different vCenter environment with completely different Cluster naming schemes exhibits this exact same behavior...

                            • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                              xtraspecialj

                              You were so nice about trying and helping Milan_Volejnik that I just wanted to come back here and let you know what the resolution was.  chrispaap came through and got me hooked up with a very good technician named Kevin who figured out and fixed the issue.  Basically, in the python script that controls these On-Demand reports uses single quotes around the variables, specifically the "cluster" variable in this case.  Well, if you take this code right here:

                               

                              vm.origin:vmware AND vm.powerstate:poweredOn AND vm.cluster.name:%s AND ((vm.cpuloadSumpeak.week:[0 TO %s] AND vm.cpuCount:[2 TO *]) OR (vm.memloadPeak.week:[0 TO %s] AND vm.memory:[2048 TO *])) 
                              

                               

                              and you replace the first %s sign (after vm.cluster.name:) with 'yourclusternamehere', and the next two %s signs (in vm.cpuloadSupeak.week:[0 to %s] and in vm.memloadPeak.week:[0 to %s]) with the numbers 30 (or a higher number if 30 returns no results on any of your clusters) and you are on VMAN 6.3.1.575 (and maybe other versions of VMAN) it may return VM's that are not in the cluster you specified (at least that was our issue in two completely separate VMAN instances looking at two completely different vCenter servers' sets of Datacenters and Clusters.

                               

                              Looking at the python script called OversizedVMsReport.py located at /data/var/hyper9/data/content/reports on your VMAN server (via a putty session), you'll see a line that says:

                               

                              else:
                                   cluster = "'" + cluster + "'"
                              

                               

                              Basically that line is just putting single quotes around the cluster variable so that they would show up inside of the search query, but remember, putting single quotes around a cluster name causes unexpected results.  What Kevin figured out was to replace those single quotes with double quotes and it would suddenly work.  Another catch though: putting double quotes inside of other double quotes didn't work, so we had to escape the middle double quote in each set with a \ character in front of them.  What we ended up with was this (the \ escapes the middle double quote so that the variable ends up being seen by the reporting engine as "cluster" by the time it's sent to resolve the cluster variable to the actual cluster name):

                               

                              else:
                                   cluster = "\"" + cluster + "\""
                              
                              
                              
                              

                               

                              To be safe, we copied and renamed the OversizedVMsReport.py  to OversizedVMsReport_Corrected.py and made the corrections in there instead. 

                               

                              You do need to do a service tomcat restart afterwards for your new report to show up in the On-Demand Reporting console.

                               

                              Hopefully SolarWinds will push out some kind of hotfix or fix it in the next upgrade.  My guess is there may be some other reports that are broken by this weird bug.

                                • Re: VMAN On-Demand Reporting (and VMAN Reports in General)
                                  tomas.kuty

                                  More tech detail about the issue:

                                   

                                  Search vm.cluster.name:"BOS - PROD - App" works as you would expect. It searches for cluster named "BOS - PROD - App".

                                   

                                  However, search vm.cluster.name:'BOS - PROD - App' is not what it looks like. Single quotes are ignored, they are not valid for search expressions. And "-" character is a synonym for NOT. So, the search actually means vm.cluster.name:"BOS" AND NOT "PROD" AND NOT "App". That's why searches seemed to work weirdly.

                                   

                                  So, yes, the issue is not in VMAN application but in the report itself. Search phrases must always be enclosed in double-quote marks if there are spaces included in the search phrase. Follow previous comment to fix the issue.