cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Which VoIP troubleshooting tools do you turn to first?

Level 11

In my last post, we discussed implementing voice into a new environment, now I figured we would discuss troubleshooting that environment after it's initially deployment. Only seems natural right?

Now, due to its nature troubleshooting VoIP issues can be quite different then troubleshooting your typical data TCP/UDP applications and more often then not we will have another set of tools to troubleshoot VoIP related issues. (And hopefully some of those tools are integrated with our day-to-day network management tools)


IP SLA/RPM Monitoring*:

This is definitely one of my favorite networking tools, IP SLA monitoring allows me see what the network looks like from a different perspective (usually a perspective closer to the end-user). There are a few different IP SLA operations we can use to monitor the performance of a VoIP network. UDP Jitter is one of those particular operations that allow us to get a deeper insight into VoIP performance. Discovering variances in jitter could point to an incorrectly sized voice queue or possible WAN/transit related issues. Have you ever considered implementing a DNS or DHCP IP SLA monitor?

*Keep in mind IP SLA monitoring can also be used outside of monitoring the VoIP infrastructure, other operations support TCP/HTTP/FTP/etc protocols so you can get the user's perspective for other mission critical applications.


NetFlow/JFlow/IPFIX:

Another great tool to have in the arsenal. NetFlow is an easy way to see a breakdown of traffic from the interface perspective. This allows you to verify your signaling and RTP streams are being marked correctly. It also provides you with the ability to verify other applications/traffic flows are not getting marked into the Voice queue unintentionally. Different vendors can run their own variations of NetFlow but at the end of the day they all provide very similar information, many of the newer NetFlow versions allow more granular control of what information is collected and where it collected from.


MOS Scores:

While this one is not an actual tool itself. Keeping an eye on your MOS scores can quickly identify trouble spots (if the end-users don't report it first that is) by identifying poor quality calls.

MOS.PNG

A good old Analog phone:
Wait a second this a VoIP deployment right? What if we do have a few analog lines for backup/AAR/E911 services we find ourselves in a situation were we might need troubleshoot that analog line. Possibly for static or functionality.

Polling more specific information:

Depending on what you trying to troubleshooting you can definitely get some great insight from polling some more specific information using the UnDP:

(Some of these will only be manageable for smaller locations/deployments)

  • Number of registered phones - displayed in a graph format so you can easily drops in registered phones
  • CME Version - Specific for Cisco routers running CME, but keeping track of the CME versions could help isolate issues to a specific software set.
  • Below are a few others I have created as well, below is a sample VoIP dashboard.

Custom VoIP Pollers2.png

29 Comments
Level 14

I like VOIP & Network Quality Manager.  I have also used NTA and Cisco NAMs to analyze VoIP traffic.  The NAMs actually show a lot of information and break out VoIP calls and traffic nicely.

Level 11

Right now, we turn to VNQM.  Its really our only tool in the area of VoiP at the moment. 

Level 11

......AAAAAAAAaaaaaaaaaaaand bookmarked for later.

Level 17

Though this is not exactly my area, IP SLA / Jitter Type Analytics and then MOS Scores tell us how well the quality is maintaining.

Normally it is when we see RED on a larger part of the setup that causes us to motion to the Telecom Engineer's.

If I could only get credential info (service account and snmp setup added to/for) our voice equipment, I would be showing off more than normal.

Level 11

You know I probably have to re-visit NAMs. I haven't looked at a NAM in quite a few years. They've probably gotten much better since the last version I worked with them

Level 15

I'm personally a huge fan of the VNQM product. Generally, the clients I have worked with use either IP SLA OR NTA. I of course try to refocus their attention on using a multitude of tools instead of depending on a single. But, to be honest, I have never really thought about using UnDPs... (I'm a little ashamed to admit that )

That being said, I am really digging the idea and the wheels are a turning! Thanks for the post!

Level 11

I'll upload the UnDP's in my post to the content exchange to get you started 🙂

Sometimes when/if I have spare time at work I fire up the MIB Browser and just go searching a particular part of the tree to see what's there. It's surprising how much good information is in there just waiting to get polled.

Level 12

we have it broke into two parts myself as a network monitoring guy have eyes on VNQM but our telephony group has eyes on nam's

Level 9

We currently use a couple of tools to monitor VOIP traffic although there is not much at this time.

Level 11

Do I lose if I say wireshark?

MVP
MVP

We have VNQM but not using it to it's full potential.

Level 12

Shortel (the phone system) offers their Director Software.  It has several troubleshooting tools as part of the application.

Level 15

Oh spare time... I do miss her.

Level 11

O wow, I can't believe I forgot about wireshark!

And to think I've used that quite a few times to listen to VoIP calls to troubleshoot!

I had an interesting issue in the past where VoIP conversation would sound crystal clear in the capture but sound 'staticey' to the user, turned out to be a software caveat with the PVDM modules.

Level 13

It seems our telecom group will look at stats like the MOS and reach out to us in the NOC to check network congestion and QOS settings if they think it's network related.

Level 9

Most of my customers/users are not using VNQM as they should be, I frequently try to help them set the tool in the right way but, still much way to go...

Level 10

We use VNQM.

Level 12

VNQM and Riverbed SteelCentral, (formerly Cascade Express) for Netflow aggregation & Application Monitoring, and Alerting.  Does constant Wireshark Caps for replay and analysis including VoIP using Pilot.  Very nice tool integration!

Level 11

I had demo'ed a few Cascade Express boxes & Cascade Pilot earlier this year, I was really impressed with them definitely a great application performance tool!

Pilot was also extremely useful definitely made it easier to go through pcaps, and grabbing pcaps directly off appliance was a nice plus.

MVP
MVP

just done the same as jaimeaux

We are not there yet, but following these discussions with interest for when we need it!

Level 11

i haven't had the opportunity to use net flow, I would like to try it out.

Level 9

No, you get extra points.

Level 9

There a lot of good choices to go with here.  My answer is simple: Whatever tool gets me closest to the answer in the shortest time.  No, not a blow-off answer, it's just a matter of selecting the right tool for the situation.  In a NOC setting, there isn't a single tool or method that is the go-to.  You know the possible areas and use the tool that is linked to the most likely culprit of the issue.

Level 10

And if all of this tools are not enough, a very good knowledge of debug commands are soooooooooooooooooo IMPORTANT

Level 13

Stephen Occhiogrosso you mentioned RPM early but didn't comment any further / provide specifics (being Junipers take of IPSLA).

As it's not out of the box for VNQM, can you share and advise how you have gone about and implemented your monitoring for it - is it using VNQM somehow, or back to UnDPs from NPM to get the metrics.

Trying to get them discovered initially from NPM and reported correctly with 13 metrics per site was a huge pain for us - Achievable, but a pain none the less, so if there is another way your able to advise that is easier I'm keen to give it a try.

TIA. Rgds

Shane

Level 7

made my day

Level 15

I have used Wireshark in the past. 

MVP
MVP

csim start <phone number> for CME was quite helpful when I last needed it (2006).  Assuming it's still an available hidden command in Cisco IOS.  Performs call setup and rings a phone (assuming the number is reachable).  (Also a really useful command to drive your coworkers nuts.)

Cisco IOS debug commands were also quite useful for CME troubleshooting.  ISDN Q921 and Q931 come to mind.

Level 15

Interesting discussion and good article.