Virtualization Manager is built on search - all the widgets, alerts, charts, etc. are simply queries that bubble up the most important data to your fingertips. This allows us to quickly alert you about potential problems, including configuration recommendations for over and under allocation of resources (like CPU, memory, storage, etc.). Many users have asked about how these recommendations are calculated, and the answer is - its all in the queries! And even better - you can easily copy and edit the alert queries to build your own.
I was recently working with a customer on the alert recommendations for the default alert "VM Memory Overallocated". The queries for this alert are based on average memory usage over a week, and the customer was concerned (and rightfully so) that if he followed our recommendations, he might induce swapping during memory peaks. So I took a deeper look to see what we could do.
Make a copy of the current alert
First, I took the current alert and made a copy to work with:
- From the Administrator Dashboard, scroll down in the All Alerts widgets and click "VM Memory Allocated".
- At the bottom right of the screen, click "Configure".
- At the bottom right of the screen, click "Save As", enter a new name and click "Save". I changed mine to "VM Memory Peak Overallocated".
- You should see the new name at the top and now click "Configure" at the bottom right again.
Change the scope
I decided to change the scope, editing the last item to evaluate the weekly peak instead of the average and increased it from 30% to 50%. I wanted to make sure I captured any VMs whose peak memory utilization was under 50% of the allocated. Here is the query in full (bolded is the part I changed):
vm.powerstate:poweredOn AND vm.memory:[2048 TO *] AND vm.memloadPeak.week:([* TO 50])
So this establishes the scope as all VMs with at least 2GB of RAM whose peak memory usage did not go above 50% for the entire week.
Change the recommendation
Now that we have the scope of the query set to the VMs we are interested in, we next tackle the recommendation. In this case, I am going to take the peak utilization and multiply it by the VM Memory to get a peak utilization. I will multiply that by 1.25 to add some buffer, and then do some fancy xpath functions to make recommendations in multiples of 256KB.
concat ('Reduce ', /virtualMachine/memory, ' MB to ', (ceiling(/virtualMachine/memoryPeakUtilization/week * /virtualMachine/memory * 1.25 div 100 div 256))*256, ' MB')
Just to break that down:
- Multiply weekly memory peak * VM memory * 1.25 (buffer) div 100 (because the peak is given as a percent)
/virtualMachine/memoryPeakUtilization/week * /virtualMachine/memory * 1.25 div 100
- Build a multiple of 256KB by dividing by 256, rounding up to the nearest integer, and then multiplying by 256.
(ceiling(... div 256)) * 256
Save and view the results
Once you click "Save" it will return you to the alerts view and immediately show you all VM's that match this condition. In this case, we now have 31 VM's that meet the criteria and now have memory recommendations for them.
To validate the recommendation, click on the scroll icon for one of the VM's in the list. This will take you to the detail view of the item. Switch to the XML by clicking the very small icon next to the name, this will allow you to see the latest instances of all the data.
Once you have the XML View open, you can look for the corresponding metrics (via the search bar at the top) to validate the suggestion:
- memory: 2048
- memoryPeakUtilization: Week 19.99
Calculation: 2048*19.99*1.25/100=511 so rounding up to the nearest multiple of 256 is 512MB, which matches the recommendation in the image above.
So in this case, we could reduce memory on this VM down to 512MB and have a 25% buffer over the historical peak value.
Additional alert features
Some of the features of alerts we skipped over:
- Condition needs to be sustained for a certain amount of time.
- Notification via email - we can send these alerts to multiple users
- Execute an external action - for example, capture the currently monitored processes on the VM
- SNMP traps and overrides.
For more information on alerting, please see our help documentation.
Alert Ideas (and more)
The default alerts packaged into Virtualization Manager are based on lots of customer feedback, but they are by no means a one size fits all! We highly recommend you customize the alerts (and dashboards and widgets) to fit your environment. For example, the choices I made on the building the above alert (ex: weekly peak 50% of total or the 1.25 buffer) should be customized to fit how you want to run your environment. Some more ways you can leverage the search:
- You could also take the searches and functions and rework the "Top-N: VM Memory Overallocated" widget on the Sprawl Dashboard.
- If you wanted to alert when a drive goes under 700MB, you can take "Guest Storage Space Alert", copy and change it to:
- Scope: vm.powerstate:poweredOn AND vm.vol.freeSpace:[0 TO 734003200]
- Alert: string-join(for $fullVolume in /virtualMachine/diskVolume[freeSpace<=(700*1024*1024)] return concat($fullVolume/mountPoint, ' (' , floor($fullVolume/freeSpace div 1024 div 1024), 'MB)'), ', ')
The possibilities are truly limitless with the search and xpath capabilities.
Other thoughts and references
Here are a few other thoughts to consider when building alerts, as well as some additional references:
- Check out our help for Search in our documentation.
- Here is a list of all properties and metrics that Virtualization Manager collects.
- Performance and configuration data are collected on different intervals, so if you are mixing them in formulas, just be aware that the times will not necessarily align.
- Rollup for peaks and averages are calculated a couple of times a day and restart at the beginning of their time period (day, week, month). See this KB article for more information on data collection and rollup.
- xPath functions are fun - see this excellent general xPath tutorial.