A Series of Unfortunate Events: Using Orion's Syslog Server to Monitor VMware Events

In network management, events are bad news.  At least the ones you care about are.  You can spend your whole day sifting through various informational events, but most of the ones that need your attention are telling that something is wrong.  Most network systems share their events through Syslog, and it’s generally understood that monitoring those Syslog streams is a best practice.  Based on the conversations that our PM team has had with users of VMware, it’s much less widely known that you can monitor VMware events using Syslog.  VMware host servers support sending their logs via Syslog, and Orion has a built-in Syslog Server, so it’s a plug-meet-socket situation. 

VMware has some good docs (see also here) on how to configure Syslog on a VMware box.  It’s pretty straightforward as these go, so I’ll assume that you can turn on Syslog and point it at your Orion server.

Once you are collecting Syslog from VMware, what do you do on Orion?  Actually, you don’t have to do anything and the Syslog messages will still be collected and displayed in the Syslog View on the web console.  Doing nothing is pretty lazy, though, and if you’re bothering to read this, you’re better than that.  To get more value from that Syslog stream, you should create some Syslog rules that will trigger an email or some other notification for key events.  To create a Syslog rule, go to the Orion server and launch the Syslog Viewer (Start > All Programs > SolarWinds Orion > Syslog and Traps > Syslog Viewer)

 

image

Select View > Alerts/Filter Rules… and you’ll see this:

image

Then click the “Add New Rule” button to see this:

image

You can get the complete steps for configuring Syslog Rules from the Admin Guide.  The rule system is rich, so you can narrow down your alerts based on IP address ranges, hostname pattern, and message severity, which will allow you to get an alert on errors in hosts with one naming convention but not another.  Most importantly, for VMware Events, you can create rules based on the content of the Syslog message.  The engine scans the content of the message and checks it against the string or regular expression you type.  For instance, if you want an email any time a VM is powered off, you can create a rule that triggers when a message with “*is powered off” appears.  Similarly, to alert when a VM migrates from one host to another, alert on “*migration*.”  Look through you Syslog messages to find the key phrases for your rules.

 

image

When the rule conditions are met, you can take any number of actions, including the usual send an email, but you can also do some Syslog-specific actions like tagging a Syslog, modifying the Syslog message, or  forwarding the message on as a Syslog or converting it to an SNMP Trap. 

image

What events should you alert on?  Some of the more common cases we’ve seen are VM cloning, VM migrations (i.e., VMotion), taking snapshots, deleting a VM, and changing resource allocations.  The exact set will depend on what matters to you.  To create those alerts, just start collecting Syslog and then scan the logs for key messages.  As you find ones that seem important, create a rule.  You don’t need to do it all at once—you can create them over time until you’ve tuned the monitoring to fit your environment.

Thwack - Symbolize TM, R, and C