Latency reporting both short and long-term

We have a few hundred nodes that are spread out across the country and connected by ISPs that we do not directly manage. We would like to monitor both short term latency and long term latency (for trends).

For short term latency, based on 60 second polling intervals from the Orion engine, I'm using this SWQL:

SELECT TOP Caption, NodeID, AvgResponseTime, MinResponseTime, MaxResponseTime, PercentLoss
FROM Orion.Nodes
WHERE AvgResponseTime > 200 OR MaxResponseTime > 1000
ORDER BY AvgResponseTime DESC

That's fine for the last five or ten minutes, but I'd also like to be able to observe trends based on an hourly or daily basis. For example, if a node was showing an average response time of 250 ms yesterday but is now showing an average response time of 1500 ms, I need a way to flag that. Not necessarily an alert, but something to put in custom html widget would be fine.

We only have NPM.

Any ideas?

Find more posts tagged with

latency

Accepted answers

adam.beedell

Aah you've entered into a bit of the tool which is doable, but hard to think about. I have lots of stuff in this place which has been an open question in my head for a long time.

There's a very very cool post in here on "Alerting on volume growth" by @Seashore which is in this space i've never fully digested but have bookmarked.

There are different ways to approach this, but i've not got a generalized solution, he's some possible components

Work out cool slope and delta stuff like seashore
Use some calculated macros like ${BASELINE} and ${STD_DEV} to refer to "normal" in one way or another
often if you have a latency change from 200 to 1000 it's off the back of routing changes you can detect in stuff like OSPF/BGP sections of the tool
If you use either a custom property, or the Warning/Critical values as a store of "normal + x" (you can write to these as an alert action, or programatically), you can compare to that in later code/alerting/etc
Using a node/interface as the source isnt necessarily the representation of a whole path, people for sure use netpath for this, or in a Cisco environment IPSLA, or in SAM locally placed application monitors running ping or trace checks. I think people tend to do stuff in this space manually and update most frequently. As above you could scale it up given an update loop.

adam.beedell

Download SWQL studio and have a poke around, if you're querying this stuff historically you need to get your heard around the Weight value and Window functions. Weight is like ~how much time is this timestamp worth and Window Functions are like ~Create buckets for each Hour/Day/whatever (and do math to stuff on the way into the buckets)

All comments

adam.beedell

Aah you've entered into a bit of the tool which is doable, but hard to think about. I have lots of stuff in this place which has been an open question in my head for a long time.

There's a very very cool post in here on "Alerting on volume growth" by @Seashore which is in this space i've never fully digested but have bookmarked.

There are different ways to approach this, but i've not got a generalized solution, he's some possible components

Work out cool slope and delta stuff like seashore
Use some calculated macros like ${BASELINE} and ${STD_DEV} to refer to "normal" in one way or another
often if you have a latency change from 200 to 1000 it's off the back of routing changes you can detect in stuff like OSPF/BGP sections of the tool
If you use either a custom property, or the Warning/Critical values as a store of "normal + x" (you can write to these as an alert action, or programatically), you can compare to that in later code/alerting/etc
Using a node/interface as the source isnt necessarily the representation of a whole path, people for sure use netpath for this, or in a Cisco environment IPSLA, or in SAM locally placed application monitors running ping or trace checks. I think people tend to do stuff in this space manually and update most frequently. As above you could scale it up given an update loop.

Chris-S

Thanks Adam, that gives me a lot to think about.

I can do a lot of what is needed programmatically. The parts I'm missing is what tables and fields contain latency (min max avg) values and what frequency of that data? Is it timestamped? What is the relationship between what I assume will be multiple tables?

adam.beedell

There are simpleish ways to achieve what you want, just bear in mind that different solutions will bear different fruit I suppose.

Chris-S

Thanks, Adam, I'll keep poking around in SWQL and see if I can find what I need.