Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Best Practice: Monitoring Volumes in a Cluster

Has anyone monitored volumes of a virtual node of a cluster? I ask because I think that is probably what I need to do...

Today I was notified by a DBA that they did not receive any alerts on a cluster volume which reached 95% full. I know that I had added the volumes to the single node in the cluster that owned the volume at that time. I think that this is where I went wrong. My best guess is that when a volumes fail over to a different node, solarwinds will not recognize this and so the volumes will no longer be monitored. I'm just looking for some feedback and results of what others have experienced.

Any help would be appreciated.

Find more posts tagged with

volumes

cluster

Accepted answers

FormerMember

when monitoring a volume on a cluster, you need to add a new node specifically for the cluster IP (often there are multiple cluster IP addresses, so you need to add a new node for each of them)

You need the cluster admin dude to give you an IP address, and the volumes presented to the IP address.. add each IP to Orion as a new node, and when it discovered the volumes, make sure you only select the volumes specifically presented to that virtual node IP address

You will also need the virtual cluster name, as Orion will pick up the name of the Physical node currently hosting that virtual node/IP, as that is what SNMP returns (not Orion’s fault)

I do this a lot and it works a treat.

you will however, need to check it every now and again, as Orion can still get confused during a failover, and it looses communication with the volume and never recovers. you have to un-manage the node, and re-manage it again for it to pick up monitoring again (not sure if this is fixed in 9.5.1)

cheers

dan

All comments

FormerMember

when monitoring a volume on a cluster, you need to add a new node specifically for the cluster IP (often there are multiple cluster IP addresses, so you need to add a new node for each of them)

You will also need the virtual cluster name, as Orion will pick up the name of the Physical node currently hosting that virtual node/IP, as that is what SNMP returns (not Orion’s fault)

I do this a lot and it works a treat.

cheers

dan

TGhosh

I usually add the actual IPs for each node (primary and secondary), Add all the resources respective to each node (usually the "active" node will have all the volumes listed out while the passive node will not)

Then, I add the third node as the cluster node, using the SQL cluster Ip address. Not the MSDTC cluster address or the MS Cluster Ip address.

reason being, the APM Template for SQL server works only when testing against the cluster node when its using the SQL IP. Or atleast this is what I've found by trial and error.

the other piece of advise I have is to make sure you are monitoring the network interface that is labeled "Microsoft Failover Virtual Adapter"

with every cluster event, I always should get alerts about the volumes disappearing on one node AND an interface alarm from the "Microsoft Failover Virtual Adapter" going down. In some scenarios where the cluster didnt failover cleanly, we saw that the Failover Virtual Adapter did not come back online.

jrich

I am curious how you were able to discover the VIP via SNMP in order to add the disk resources in NPM to be polled. Reason I ask is because we have tried this on numerous occasions in Windows 2003 and Windows 2008 and now R2 and we have been unsuccessful discovering the VIP via SNMP. I am able to ping the device from my solarwinds server just fine and have ensured that firewall rules are properly in place for the subnets these clusters reside on.

I am able to poll the two physical nodes and discover them by SNMP just fine as well in NPM. The issue I am having is that I would like to use NPM to poll and monitor the cluster disks.

I can do this by adding the disks to the primary node of the cluster but if a failover occurs then the second node will not see the disks (even if I have failed over the cluster, added the disks and then failed the disks back over to the first node, I would still have to go back into NPM and list the resources and re-check the disks and submit the changes.) Since this is a manual process this would take manual intervention in the monitoring portion in order to monitor any cluster with any shared volumes.

Does anyone have any recommendations on how to run a discovery of the VIP and have it answer back via SNMP?

kb1lxm

jrich,

Yeah I ran into an issue with monitoring cluster Virtual Resources behind a firewall. Here's the deal, when you send an snmp query to a cluster resource IP, it will get picked up on that IP. However because of how the binding works the SNMP Agent on the cluster node is what will reply. So we have a clustered SQL Server for example and we have the cluster resource named CTSQL01 and we have two cluster nodes named CTCluster1 and CTCluster2, here's how it would look:

Orion Server------SNMP Query--------> Firewall ----------> CTSQL01 (Running on CTCluster1)

Orion Server (SNMP Response Dropped) Firewall <---------- CTCluster1

The firewall can't tell that the query to CTSQL01 and the response from CTCluster1 are the same conversation. So the only thing we could do is open up all high random ports (ick) to our Orion Server so SNMP responses could pass right through.

This is all based on how Windows Clustering works and unfortunatly there is no way to fix that behavior on the cluster itself. Hope my response is clear.

ichoudhury

I'm having the same issue as jrich with a W2k8R2 Cluster. Firewall is not the issue for me and the problem is due to who is listening to the SNMP port. So if I use the portqry tool to check on it

portqry -n <w2k8r2_Cluster_N1> -e 161 -p udp -cn !snmp_CS!

I get the response just fine but not when I do this

portqry -n <sql-instance-vip> -e 161 -p udp -cn !snmp_CS!

(Regardless the above sql-instance-vip is active on Node1 at the time of my test)

So you wonder, if the SNMP port is being listened by the Node IP? That would be the culprit of that type of behavior. But I was wrong

UDP 0.0.0.0:161 *:*

(Above means that any IP in that specific instance will answer for query for 161)

So my trace shows that it is actually passing FW from source where I am doing the portqry, but server isn't responding back .. just only responding back when I specifically using the server (Node) IP.

So possibly, SNMP and Cluster service start sequence probably playing a role? One has to be started before the other? Oh wait a minute .. only IP that is responding back to the SNMP query that is bound to the NIC and the cluster VIP or the SQL instance VIP are configured in the cluster manager ... they are coming into the picture after SNMP service already up and running ...

I think I am on to something :-p

kpieckiel

I'm having similar issues as others, except I don't have a firewall involved. No physical firewall exists between my Orion server and my SQL Server cluster, and I turned off Windows Firewall to ensure that wouldn't get in the way. I can monitor physical nodes and drives attached to those nodes but my SQL resource doesn't respond to any SNMP requests, much less provide me with the list of drives to monitor for the SQL resource being provided. What am I doing wrong here?