How to Deal with High Latency Coupled with High Bandwidth Usage

I have a few hundred sites to monitor. More than a few are on slow MPLS connections (1.5 Mbps, for example).

When someone starts some legit traffic usage, we get a bandwidth alert. Cool. That's what we want. What we also get is high latency at every device at the site where the download is taking place once they inevitably go over 250ms. All of these devices generate alerts, which generate tickets, which need to be looked at and closed/merged, and this happens multiple times per day. I'm talking 300-400 tickets per day because there are a lot of sites.

This high latency alert has essentially just become a noise maker that goes off every few minutes, and people are learning to just ignore it. What suggestions can be made to help cull this to a reasonable number? Can I do something like "Alert unless bandwidth is high", or have parent/child relationships setup (Its own nightmare that isn't currently setup. At all.) that won't show high bandwidth if the parent device does? Does the parent/child relationship even cover that? I can only reliably find information that shows it only has an impact on child devices when the parent is in a down state.

I don't want to just turn the alert off because it's effectively useless in our network, but I'm starting to think that's my only realistic option at this point.

Find more posts tagged with

Accepted answers

rschroeder

I recommend studying some of the technical publications and Video sessions created by Alert Master @adatole. In my opinion, he's the "fine" in "fine-tuning alerts" for Solarwinds, and I live by his philosophy about an alert not being useful if it isn't actionable.
It's worth creating the parent-child relationships even though it may be a large and tedious task. I did that and my team greatly appreciated not being inundated with alerts from nodes that actually were caused by parent nodes or WAN latency problems.
You may find it beneficial to leverage Netflow Traffic Analyzer's capabilities to track and graph issues with latency associated with uplink congestion. And I mean "beneficial" as in using the data to prove to upper Management that the small pipes are affecting business through congestion.
Use the decreased performance abilities of sites that are impacted by WAN congestion as justification for getting bigger pipes. Maybe contract with ISP's or WAN providers to give you burstable service that allows temporary larger data flow at discounted rates.
Use your NTA data to help craft firewall rules, router flow limiting rules, and I.T. policy such that only specific sites, approved to be white-listed by Management as "business critical", can be accessed under times & conditions your I.T. superiors can define.
Investigate SD-WAN services and bring their costs and benefits to the attention of your Management. Maybe you'll find they'll support having additional links of other kinds of service (broadband, T1's, Ethernet, 4G or 5G, etc.) that will only activate under specific situations. Your business data traffic must flow swiftly and reliably. If that flow's uplink pipes were not sized appropriately due to underestimating the data flow or growth, or cost was justed to justify limiting their size, the data collected can be a way to justify having other paths for the data to get where it needs to be in a timely manner.
Use your NTA data to implement QoS and CoS more effectively. If your data lines are also being used for VoIP and Video, you should be able to easily identify that (along with complaints from users of those services at the remote locations) as justification for fatter pipes. But if the budget isn't forthcoming, you can give Management the opportunity to make the decision about what's more important: getting the data between two sites, or having effective video meetings and telephony that isn't interrupted.
Be thinking about how DMVPN can be a good solution for a hub-and-spoke network where spoke members may need to talk directly to each other, instead of them having to both rely on the center hub. It can save bandwidth on those uplink legs--IF your sites are spending time talking to each other instead of solely talking to the center hub.

Yes, there's a lot to think about. Some is simply technical, and that's probably the easy part. Yet it may still be insufficient, and you'll have to go up into Layers 8 and 9 (budget and politics) to have the policy makers free up funds for the greater good, or make the hard calls about what simply can't be done at all versus what is acceptable to do slowly. They'll need to think about how your customers are affected by the congestion, and how other companies may be leveraging that slowness against you to steal your business away.

All comments

rschroeder

I recommend studying some of the technical publications and Video sessions created by Alert Master @adatole. In my opinion, he's the "fine" in "fine-tuning alerts" for Solarwinds, and I live by his philosophy about an alert not being useful if it isn't actionable.
It's worth creating the parent-child relationships even though it may be a large and tedious task. I did that and my team greatly appreciated not being inundated with alerts from nodes that actually were caused by parent nodes or WAN latency problems.
You may find it beneficial to leverage Netflow Traffic Analyzer's capabilities to track and graph issues with latency associated with uplink congestion. And I mean "beneficial" as in using the data to prove to upper Management that the small pipes are affecting business through congestion.
Use the decreased performance abilities of sites that are impacted by WAN congestion as justification for getting bigger pipes. Maybe contract with ISP's or WAN providers to give you burstable service that allows temporary larger data flow at discounted rates.
Use your NTA data to help craft firewall rules, router flow limiting rules, and I.T. policy such that only specific sites, approved to be white-listed by Management as "business critical", can be accessed under times & conditions your I.T. superiors can define.
Investigate SD-WAN services and bring their costs and benefits to the attention of your Management. Maybe you'll find they'll support having additional links of other kinds of service (broadband, T1's, Ethernet, 4G or 5G, etc.) that will only activate under specific situations. Your business data traffic must flow swiftly and reliably. If that flow's uplink pipes were not sized appropriately due to underestimating the data flow or growth, or cost was justed to justify limiting their size, the data collected can be a way to justify having other paths for the data to get where it needs to be in a timely manner.
Use your NTA data to implement QoS and CoS more effectively. If your data lines are also being used for VoIP and Video, you should be able to easily identify that (along with complaints from users of those services at the remote locations) as justification for fatter pipes. But if the budget isn't forthcoming, you can give Management the opportunity to make the decision about what's more important: getting the data between two sites, or having effective video meetings and telephony that isn't interrupted.
Be thinking about how DMVPN can be a good solution for a hub-and-spoke network where spoke members may need to talk directly to each other, instead of them having to both rely on the center hub. It can save bandwidth on those uplink legs--IF your sites are spending time talking to each other instead of solely talking to the center hub.

obi-sean

I'll definitely take a look at those videos. Realistically, if they don't do wonders in paring down the alerts: It goes to layers 8 and 9 to look at ISP-related infrastructure changes, which can take months. Assuming we get to that point: the best answer would be to just carve those offending sites out of the alert, or just disable it so we don’t become numb to these alerts and their severity when not directly coupled to a bandwidth issue.

I had a feeling this is what I would be facing, but I wanted to exhaust as many options as I could before that.

Thank you for your reply to this. It’s much appreciated.

Edit: Parent/child is on the list of things to do. My mention of it was just wanting to verify if it should be greatly upgraded in priority.

rschroeder

Parent-Child relationships will prevent alerts from every Child object when its dependent Parent object has issues. That can help you a lot, quickly.