Our company's network was recently experiencing issues with device connectivity. The issues ranged from VOIP phone failures, to wireless scanning equipment unable to connect to the network. After weeks of research we were able to pin down the culprit - Dameware. (Story below; TL:DR = Broadcast traffic of ICMP IPv6 from Dameware hurt the network) Please see questions at the end.
Sunday - It all started after we were hit with a severe virus that spread through the company touching almost every PC we have. To remedy we had to "lock down" our system-to-system security. This meant it would be very difficult to install an application from one PC to another and this is how we would typically use Dameware with the HelpDesk team. To circumvent, the decision was made to push out Dameware to every PC using SCCM and group policy. We now had remote support on all of the PC's and would not have to worry about issues going forward. GREAT!
Monday - The calls start to trickle in about various network related issues. Our users are the typical kind; most don't complain unless it completely stops them from doing their job. So one or two complaints the first day. We examined but could not find anything obvious. It was strange, but we kept searching.
Tuesday - 4-5 more sites complaining of issues, no real root in site. We carry on troubleshooting. Switches report next to zero % usage and most devices are working at these sites. The most issues we see come from the wirelessly connected devices, but our wireless system appears perfectly fine on the service.
Wednesday - another 5-6 sites reporting issues, it is full fledged network issue. We keep searching, we think possibly virus related or some new group policy we pushed. Wireshark turns up mostly normal packets, though we do see a lot of ICMP IPv6 hits.
Thursday - Out of hand now, boss wants answers and soon. We call in outside support vendor. We have never experienced this type of problem before and apparently our troubleshooting skills are lacking.
Friday - Holiday weekend coming up, users are out of office, not that many complaints. We think the issue has subsided. We speak with the vendor anyways to set a game plan for the following week just in case.
Sat-Mon - Holiday weekend, users off celebrating the end of summer. No worries, but I will tell you I didn't sleep well Monday night.
Tuesday2 - Phones explode with continual issues. Support Vendor makes site visit to one of our corporate offices; they can't find anything obvious. The idea to roll back Dameware as a shot in the dark was suggested. We would try it the next day.
Wednesday2 - While monitoring the network we roll back the install of Dameware on 50 PC's. Minutes later the network at this single site begins to operate normally. WTH? Reinstall back to 50 PC's, problems return. We monitor packets with wireshark and notice a large reduction in ICMP IPv6 when Dameware was removed. We do a percentage check and ICMP IPv6 is making up more than 90% of our traffic. Doesn't seem right or why that traffic is coming from Dameware.
Thusrday2 - I make a visit to a plant as my colleagues scour Dameware for any setting which may be the root of the broadcasts. Then they find it........... Teredo - Peer Name Resolution Protocol (PNRP): Register Local Common Peer PNRP Name. When this is disabled ICMP IPv6 traffic on our network goes to zero. I test and confirm a site which was having devastating affects is now working perfectly normal. Our team scrambles to change group policy to remove this setting from Dameware on every single Windows PC in our company.
Friday - Check with every site, no issues. Who would have thought?
Key symptoms of this issue:
1. Wireless devices unable to connect wirelessly
2. Wireless devices connect, but can't get IP
3. Wireless devices can use 5Ghz, but not 2.4Ghz
4. Devices unable to traverse into other subnets than native
5. Older VOIP phones unable to operate
Questions we are still searching for answers:
1. Why is this feature default enabled on Dameware? How does it help Dameware in anyway?
2. Why did this affect some devices and not others?
3. Even though this was essentially a broadcast storm in terms of packet percentage, it wasn't hurting our network device usage. What was it about the ICMP IPv6 packets that harmed the other data flowing on the network?
4. How can we avoid this from happening in the future?
We implore feedback from any and all on this topic. Honestly, anything you can tell us from your experiences would be appreciated. I will also field any questions on the topic as I would hate for this to happen to anyone else. AMA