Availability and redundancy

dhamiltonsafe

Has anyone worked with monitoring availability in an environment with redundant systems?   For example, I have a group named "Active Directory" that contains all 4 of my domain controller nodes.   If any one of those nodes goes down, the entire group takes a hit on overall availability even though Active Directory is still fully functional and available from the other 3 DC's.    I'm looking for a way to get more accurate reporting on my groups that take redundancy into account.

jm_sysadmin

Availability is tough. Orion does some stuff out of the box, but it didn't match how my organization thinks about downtime. What I do will not help everyone, but here it is.

I should say that I ask that people mute alerts during scheduled downtime, and acknowledge alerts to stop emails.

First, we have Alerts for simple apps (if down, then alert) and alerts for systems that have high availability designed into them. The HA systems have specific alerts based on service delivery ( I need 3 of 4 servers running, or I need a https response code of 200, or whatever up/down equals for that thing). All my alerts write to the Netperfmon log, but these I consider service availability alerts, and get ' : Availability Impacted' added onto whatever other text is in the log message.

I report on downtime by the down events, filtered by that message. Other alerts get triggered but don't have that text  in the log, details for the HA systems have other events so details get sent to the teams that support it, but I report on just the system availability. Since all simple systems get that in an umbrella alert, I just need to do this when exclude and build a specific alert for an HA system. (Custom properties are your friend)

Then you need to get that back out. I tend to use SWQL and pull it into excel to finish my report for graphs and stuff, but you can dump it into the default report writer too. I limit it to the last 7 days in the query below, but if you remove that and use the report engine, you can time limit based on the start or end date columns. Some of the column there I don't use in the report, but they make my SWQL happy

SELECT

      E.Nodes.Caption AS [Device]

    , a.Name as [Application Monitor]

    , E.EventID

    , E.NetworkNode

    , E.NetObjectID

    , E.Message

    , (SELECT TOP 1 ToLocal(EventTime) AS [EventTime]

    FROM Orion.Events AS [EndTime]

    WHERE

        EndTime.EventTime >= E.EventTime

        AND (EndTime.EventType = 504 -- up

            OR EndTime.EventType = 510 -- warning

            OR EndTime.EventType = 509 -- critical

            OR EndTime.EventType = 512) -- resumed

        AND EndTime.NetworkNode = E.NetworkNode

        AND EndTime.NetObjectType like 'a%'

        AND EndTime.NetObjectID = E.NetObjectID

    ) AS [EventUpTime]

    , E.EventTime AS [DownTime]

,CASE

WHEN (SELECT TOP 1 ToLocal(EventTime) AS [EventTime]

    FROM Orion.Events AS [EndTime]

    WHERE EndTime.EventTime >= E.EventTime

    AND (EndTime.EventType = 504 -- up

        OR EndTime.EventType = 510 -- warning

        OR EndTime.EventType = 509 -- critical

        OR EndTime.EventType = 512) -- resumed

    AND EndTime.NetworkNode = E.NetworkNode

    AND EndTime.NetObjectType like 'a%'

    AND EndTime.NetObjectID = E.NetObjectID

    AND EventTime IS NOT NULL

    ORDER BY EndTime.EventTime) IS NULL THEN MINUTEDIFF(E.EventTime, GETUTCDATE())

ELSE MINUTEDIFF(E.EventTime,

    (SELECT TOP 1 EventTime

        FROM Orion.Events AS [EndTime]

        WHERE EndTime.EventTime > E.EventTime

        AND (EndTime.EventType = 504 -- up

            OR EndTime.EventType = 510 -- warning

            OR EndTime.EventType = 509 -- critical

            OR EndTime.EventType = 512) -- resumed

        AND EndTime.NetworkNode = E.NetworkNode

        AND EndTime.NetObjectType like 'a%'

        AND EndTime.NetObjectID = E.NetObjectID

        ORDER BY EndTime.EventTime))

END AS [Minutes]

FROM Orion.Events E

join orion.apm.application a on a.applicationid=E.netobjectid

WHERE

    E.EventType = 505 -- down and unknown

    and E.message  like '% : Availability Impacted'

    and E.message not like '%unknown%'

    and E.NetObjectType like 'a%'

    and daydiff(E.eventtime, getdate())<7