VMAN On-Demand Reporting (and VMAN Reports in General)

I know the VMAN forums aren't quite as busy as some of the others since there are a ton of unanswered questions on here I've noticed, but I'm hoping somebody can help me here.  Anyone?

Hi, sorry for late answer.

1) if you are getting all VMs from all clusters then please contact SolarWinds support.

2) you are right, the "Oversized VMs for a specific Cluster" report is based on combination of "VM CPU Overallocated" and "VM Memory Overallocated" alerts. However the definition is hardcoded in the report. So if you modify the alerts (either "Search Query" or "Notification value"), the On-Demand report will not be updated.

And definition of the 'VM Memory Overallocation' alert:

Search Query:

vm.memory:[2048 TO *] AND vm.powerstate:poweredOn AND vm.consumedPercentMemload.week:[0 TO 30]

- max. limit (30) of vm.consumedPercentMemload.week is customizable in the report

- Search Query in the report contains also condition to select only VMs belonging to the selected cluster

Notification:

if(floor(/virtualMachine/consumedMemoryUtilization/week div 1024 * 1.5) < 256) 
then concat('Reduce ', /virtualMachine/memory, ' MB to 256 MB') 
else concat('Reduce ', /virtualMachine/memory, ' MB to ', floor(/virtualMachine/consumedMemoryUtilization/week div 1024 * 1.5), ' MB')

The 'VM CPU Overallocation' alert:

Search Query:

vm.cpuCount:[2 TO *] AND vm.powerstate:poweredOn AND vm.cpuloadSumpeak.week:[0 TO 30]

Notification:

concat('Reduce ', /virtualMachine/cpuCount, ' to ', 
(if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 80) then 1
else if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 160) then 2
else if ((/virtualMachine/cpuCount * /virtualMachine/cpuUsedPeakUtilization/week/text()) <= 320) then 4 else 8))

I hope it will help you understand how the report works.

Thanks for the response.  I ended up putting a ticket in because it is giving us VM's in clusters different than what we choose.  For example, lets say we have clusters A, B, and C in VMAN.  If we choose Cluster A for the report, we may get VM's from Cluster C.  If we choose Cluster C it might be empty and if we choose Cluster B it might be either empty, or from Cluster A.  It is really weird...

I did find the reports in /data/var/hyper9/data/content/reports and was able to discvoer the hard-coded nature of them by reading the python script that generates the report (it's called OversizedVMsReport.py.)  I took this part out of it:

vm.origin:vmware AND vm.powerstate:poweredOn AND vm.cluster.name:%s AND ((vm.cpuloadSumpeak.week:[0 TO %s] AND vm.cpuCount:[2 TO *]) OR (vm.memloadPeak.week:[0 TO %s] AND vm.memory:[2048 TO *]))

and replaced the "%s" placeholders with my cluster name and thresholds, pasted it into the Search of VMAN and it returned the correct results, as expected.  So it seems to definitely be something weird where when we are choosing the cluster something in the web code is sending the wrong cluster name to this variable.  We have two VMAN environment monitoring two separate vCenter environments and both VMAN's return the wrong cluster no mater which cluster we choose.  What the heck??

J

Send me your ticket number and I will track on our side.

Hey, I sent you the ticket number via private message, have you had a chance to look into it yet?  It's been almost a week and we still haven't heard from one person at support yet...  As I said in my message to you, we have a VP that is breathing down my neck to get him this report and he isn't too happy about the response time so far.

It's really weird. Just some ideas come to my mind:

- are the cluster names unique and distinct?

- do the cluster names contain special characters (i.e. apostrophes)?

- is cluster A name a part of cluster C name?

- are the cluster names unique and distinct? Unique and distinct, yes.

- do the cluster names contain special characters (i.e. apostrophes)?  Dashes are the only special characters

- is cluster A name a part of cluster C name?  They do share some commonalities, see below:

BOS - PROD - App

NC - Citrix - App

NC - Prod - App

etc...

I would say this theory could be it, however If I choose Boston in the report it comes back with NC - Citrix - App vm data, and those two have nothing in common except for the App at the end...  Also, our other VMAN environment that looks at a completely different vCenter environment with completely different Cluster naming schemes exhibits this exact same behavior...

You were so nice about trying and helping Milan_Volejnik that I just wanted to come back here and let you know what the resolution was.  chrispaap came through and got me hooked up with a very good technician named Kevin who figured out and fixed the issue.  Basically, in the python script that controls these On-Demand reports uses single quotes around the variables, specifically the "cluster" variable in this case.  Well, if you take this code right here:

vm.origin:vmware AND vm.powerstate:poweredOn AND vm.cluster.name:%s AND ((vm.cpuloadSumpeak.week:[0 TO %s] AND vm.cpuCount:[2 TO *]) OR (vm.memloadPeak.week:[0 TO %s] AND vm.memory:[2048 TO *])) 

and you replace the first %s sign (after vm.cluster.name:) with 'yourclusternamehere', and the next two %s signs (in vm.cpuloadSupeak.week:[0 to %s] and in vm.memloadPeak.week:[0 to %s]) with the numbers 30 (or a higher number if 30 returns no results on any of your clusters) and you are on VMAN 6.3.1.575 (and maybe other versions of VMAN) it may return VM's that are not in the cluster you specified (at least that was our issue in two completely separate VMAN instances looking at two completely different vCenter servers' sets of Datacenters and Clusters.

Looking at the python script called OversizedVMsReport.py located at /data/var/hyper9/data/content/reports on your VMAN server (via a putty session), you'll see a line that says:

else:
     cluster = "'" + cluster + "'"

Basically that line is just putting single quotes around the cluster variable so that they would show up inside of the search query, but remember, putting single quotes around a cluster name causes unexpected results.  What Kevin figured out was to replace those single quotes with double quotes and it would suddenly work.  Another catch though: putting double quotes inside of other double quotes didn't work, so we had to escape the middle double quote in each set with a \ character in front of them.  What we ended up with was this (the \ escapes the middle double quote so that the variable ends up being seen by the reporting engine as "cluster" by the time it's sent to resolve the cluster variable to the actual cluster name):

else:
     cluster = "\"" + cluster + "\""

To be safe, we copied and renamed the OversizedVMsReport.py  to OversizedVMsReport_Corrected.py and made the corrections in there instead. 

You do need to do a service tomcat restart afterwards for your new report to show up in the On-Demand Reporting console.

Hopefully SolarWinds will push out some kind of hotfix or fix it in the next upgrade.  My guess is there may be some other reports that are broken by this weird bug.

More tech detail about the issue:

Search vm.cluster.name:"BOS - PROD - App" works as you would expect. It searches for cluster named "BOS - PROD - App".

However, search vm.cluster.name:'BOS - PROD - App' is not what it looks like. Single quotes are ignored, they are not valid for search expressions. And "-" character is a synonym for NOT. So, the search actually means vm.cluster.name:"BOS" AND NOT "PROD" AND NOT "App". That's why searches seemed to work weirdly.

So, yes, the issue is not in VMAN application but in the report itself. Search phrases must always be enclosed in double-quote marks if there are spaces included in the search phrase. Follow previous comment to fix the issue.