cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 8

Any way to detect changes in a WAN interface?

Jump to solution

Not sure if this is something SolarWinds can help us out with or not, but I thought I'd ask the community their thoughts or suggestions on how to handle this.

Backstory:  We have a Metro E connection at our HQ and several of our remote sites.  Sometime during the month of June, a change was made to the interface at our HQ end and we ended up rate shaped down from a 100mb connection to a 10mb.  Users called in, on occasion, and reported slowness.  We would check netflow statistics on the routers and see that there was very little traffic coming over the links.  So we assumed that everything was ok.  Fast forward one month later, a lot of large Windows updates chewing up site bandwidth, and a ton of users complaining about slowness.  We were able to verify, by transferring files back and forth between sites that the bandwidth was definitely not there.  We look at a history of bandwidth over the last 3 months and see a significant drop off in bandwidth mid June.  Where we would sometimes hit 40mb we were only seeing 10 across the board.  No spikes at all.

My question to the community:


Is there anything, short of someone looking at weekly reports (bandwidth statements), that would be able to tell us when this happens?  We're trying to be more proactive instead of reactive so we can avoid this situation in the future.

We're utilizing NPM, NCM, IPAM, and Netflow.

.

Tags (2)
1 Solution

You may see the speed of the interface change * AND that normally shows as a sys log entry(my experience).... so now you are into opening your sys log viewer and creating an alert based on conditions/wording for a working alert there.

Monitoring past your last node(Or in between main and remote sites) is a hard thing to do. Last thing the ISP wants to do is give you full visibility. We monitor what we can, and try to keep an eye our for any serious issues or red flashing. Did your speed change result in any increased error's or discards? If the link stayed up while the speed changed then EIGRP neighbors alert won't help... consider the full scope of what changed and either pick out the obvious triggers to show that your speed or link has changed.  From there drill down to conditions/events that you think may contribute and create an alert on those to ONLY yourself until you get it figured.

I wish AT&T used SolarWinds NPM so they could build me a view for all the circuits and remote sites we have.

View solution in original post

12 Replies
Level 8

I am assuming you contacted your network provider and verified that they are not currently having any issues? Perhaps this is a bit level one but at the same time I don't know how many times we've worked on a network "slowness" problem only to find out a couple hours later that the supplier was having issues. Depending on the location of your sites you could possibly experience downtime depending on your network providers pipe. We had a few issues regarding offices too close to suburbs during the holidays and end of school sessions. Good luck!

Hey Brian,

Yes we contacted our provider.  (3 times actually) and the first 2 times they told me the same thing..."Nope, nothing on our end is hosed up, this must be your end."

Finally, on the 3rd call, after I explained that transferring files from one remote site to the next or from one site to HQ was yielding speeds of 0.13mpbs, did the ISP actually take a deeper dive.  They found out some "legacy equipment" was swapped, on their side, and they didn't have any documentation as to what our interface was supposed to be set at, so they put it at 10Mb.   Once we got the our sales rep involved and our CIO made a phone call or two, it was back to it's normal speed within 2-3 hours of my 3rd phone call.

Thanks for the tips above guys, appreciate it.  I did see interface drops on the interface and am going to be working on a way to alert off those.  For now I'm utilizing a SolarWinds report that shows me the max bandwidth on all Metro-E interfaces and I have it running every day at 4pm.  (Sends the report to all the network admins)  A little more manual labor than I'd like, but at the same time, I don't enjoy being on the receiving end of a CIO / I.T. Director thrashing. 

Level 12

you could create threshold alerts.  max threshold and min. thresholds.

Level 8

Using NCM, we have a daily job scheduled to download a config and cache if it has changed from the previous. However this obviously would not work if you are not an admin of the device.

You should have your interface drops JUMP through the roof.    That would have / should have been an indication of something is not quite right.   You can also use NCM to track changes and get reports / alerts ect when this information is changed on an interface.  

Level 17

1. You can highlight the text on a trap when your interface is changed and alert on that sequence from the Trap. (Trap Viewer on your main server)

2. Real Time Config Changes Notification. It would have sent you an email at least, and you could have referenced the link to 'see' the change that was made.

3. Interface Alert - Max Recv/Xmit Traffic Rate Today(use ANY of the following apply) is less than or equal to (Threshold to alert on) 10 Mbps    ** this one may give false positives **

Thanks for the quick response.  Unfortunately we do not have access to the ISP Managed device to do any interface monitoring/reporting.  So I fear that makes point 1 and 2 moot.

I thought about option 3, however I know that we're going to get a ton of false positives.

I honestly don't think there is an easy "automation" answer.  But I'm open to suggestions.   Even if it means another software solution.

0 Kudos

I arranged with my managed routers from my ISP to feed me traps and I am able to poll for SNMP. Contact your ISP and see if they are willing to setup a RO community for you.  I even talked them into giving me netflow and a simple menu on the routers.  Most ISP's will work with you to better your monitoring and alerting environment.  If all you are worried about is configuration changes, then config traps is the minimum and I don't see why your ISP wouldn't send those to you!

0 Kudos

Alert on the interfaces that feed the circuit. If you get alerts on your own interfaces call your service provider to check the line (non intrusive of course unless the circuit is already down).

I have called one of our ISP's before and stated I am seeing errors and I need to figure out if this is on my side or yours.

*But your's is mainly about throughput * so in your story = Sometime during the month of June, a change was made to the interface at our HQ end, was this change on your equipment, or was this something the ISP did?

If your Equip - Config Change

If ISP Equip - then you need a Trap Alert for interface speed/negotiation changes  (in place of, or in conjunction with your lack of speed trap)

This was on the ISP equipment.  I probably didn't pick the best wording, when I said "at our HQ end"  I meant this was on the HQ Metro E end and not on a remote Metro E connection.  All the rest of the MetroE interfaces appear to be left at their normal bandwidth.  10mb, 20mb, 10mb etc etc.

My question is if the ISP is doing a rate limit, would we necessarily see any negotiation changes?

0 Kudos

You may see the speed of the interface change * AND that normally shows as a sys log entry(my experience).... so now you are into opening your sys log viewer and creating an alert based on conditions/wording for a working alert there.

Monitoring past your last node(Or in between main and remote sites) is a hard thing to do. Last thing the ISP wants to do is give you full visibility. We monitor what we can, and try to keep an eye our for any serious issues or red flashing. Did your speed change result in any increased error's or discards? If the link stayed up while the speed changed then EIGRP neighbors alert won't help... consider the full scope of what changed and either pick out the obvious triggers to show that your speed or link has changed.  From there drill down to conditions/events that you think may contribute and create an alert on those to ONLY yourself until you get it figured.

I wish AT&T used SolarWinds NPM so they could build me a view for all the circuits and remote sites we have.

View solution in original post

I'll take a peek at the sys log viewer and see what I might be able to use in there.

I figured monitoring in between would be difficult.  As far as I can tell, the speed change didn't result in many discards, there were some output discards, and it's something I had thought of setting up a monitor for.  I haven't checked the discards since, but it's something I'll keep an eye on.


Thanks for the suggestions on this!  much appreciated.