Active-Passive Monitoring Script

Hello,

At my organisation, we have several systems setup in an active-passive configuration where in the event of a failover, the passive server will become active. We are monitoring the application on both servers, but on the passive server, the services are showing as down.

Is there a way to setup alerts so they will only generate if services are down on both the active and passive servers at the same time?

Kind regards,

Luke.

Find more posts tagged with

sam

Accepted answers

All comments

jm_sysadmin

Yes, in the alert trigger set it to require all conditions, and set both sides as down.

brscott

Above is the direct answer to your question and probably where you should start but.... I am going to describe a different philosophy on monitoring and alerting. Keep in mind that every situation is different and your question is generic. I am going on the assumption here that you want one primary server to be the active most of the time and and passive secondary server that only takes over when the primary is not available. I am making that assumption because of your wording in the question. Also making the assumption that this setup has a shared IP address that moves between the "active" server.

Setup 3 nodes in orion. MyPrimaryServer1, MySecondaryServer1, and MyCluster1.
MyPrimaryServer1 and MySecondaryServer1 are both setup like normal, full monitoring of services, disk, whatever
Add Mycluster1 using the cluster IP address and ICMP only
Also add SAM monitors on all 3 nodes for whatever you have. This could be port monitors, 443(https), 1433(mssql). It could be sql select query, whatever. This monitor (or even several monitors) should tell you that your application is up and running correctly. Let say for example that you have a web server application. You setup a URL monitor https://${IP}:${PORT}/app/login.html
Now you have a LOT of information you can alert on. For example:
1. You can alert on the status of the http monitor on MyCluster1. 24 hours a day 7 days a week, and in a short amount of time.. you want to know if the system is down, because it's important, and you want to know immediately.
2. you can alert on http monitor on MyPrimaryServer1, but only during waking hours, say 8am to 8pm, 7 days a week. It's important, but not important enough to get woke up if the secondary took over successfully. You know the secondary took over because the monitors on MyCluster1 is not alerting.
3. No alerts on the http MySecondaryserver1. It's just for your information. You still alert on all of the standard server stuff just like you would any server, such as disk space, CPU, or memory... whatever you feel you need.
4. If you have planned maintenance then you mute the alerts on MyPrimaryServer1 for the expected time, and do the maintenance. If all goes well, then no alerts created... but during this maintenance window, if the secondary has an issue and the application crashes, you still get alerts because of the monitor on MyCluster1.
5. These are just examples, the real point is to monitor the cluster as a separate node from the member servers to give you these options.

The next step is to make generic alerts instead of specific complex alerts. Use custom properties. Yes you can create a complex alert like if X and Y is down, then alert this way. If Z is down, alert this way. If A or B is down alert this way. BUT Instead create a custom property, in the above example lets say the custom property on the SAM http monitor I mentioned. Lets call the custom property AlertingHours and it can have a list of values, such as 24Hours, WakingHours, BusinessHours, or Never.

Now create an alert where ANY SAM monitor is down and has a value of AlertingHours=24Hours, then alert (with no time restrictions). If any SAM monitor is down and the value of Alertinghours=WakingHours, the alert but only from 8am to 8pm/ 7 days a week. If its down and has the value of Alertinghours=BusinessHours then only alert 8am to 5pm, Monday-Friday.

Now you don't have to keep creating alerts, you just assign that custom property to any monitor and BAM it's done. Yes it will quickly become more complicated than that... custom properties for everything, you will have to join CPA, Custom properties anonymous. We will assign you a custom property for where you are at in our 12 step program. And yes you will later wish you had done all of the custom properties completely differently than you did... this is covered in step 5.

mrldg123

Hi,

Thank you for the solution. I found another way to do it as well. I have put the monitored applications in a group, and then alerted on the group when availability is 0%.

Kind regards.

adam.beedell

This reply should have more love!