Version 1

    One day back in February of 2014 I saw odd Internet bandwidth utilization patterns as I was watching NPM.  My corporate Internet bandwidth should have been ~75 Mb/s, and it was ramping higher and higher--much more than ever before. 

    I was seeing >350 Mb/s, which continued ramping up until it reached 800 Mb/s--all on a 200 Mb/s Internet pipe.


    NPM was the single-pane-of-glass that showed me the bandwidth utilization, increased latency, and firewall syslogs.  The syslogs prompted me to SSH in and run real-time showaudits on our firewalls. which showed a small amount of incoming TCP 123 traffic corresponding to a huge amount of outbound traffic on TCP port 123.


    A quick Google suggested we were receiving a reflection attack from the Internet.  A support call to our firewall vendor confirmed it was the NTP-123 DDOS attack; a steady flow of small inbound packets were generating (reflecting) enormous outbound responses from us destined to a specific site on the Internet.


    The reflection was targeted to bounce off "victims" and participate in a DDOS attack against a South American business.  Not only were the all the other accumulated DDOS participants unwittingly attacking that business in South America, but our unwitting participating was heavily impacting my corporate Internet in the process.  There was virtually no bandwidth left to handle our inbound/outbound traffic.


    It wasn't long before the Help Desk was calling, and my boss & HIS boss were in my cube watching Bandwidth Gauges track and graph our Internet bandwidth utilization hitting 800 Mb/s on a 200 Mb/s service.  In the mean time I configured a discard on our external firewall, but the traffic was still incoming from our ISP.


    Soon I had the ISP on the phone, and I asked them to filter out the NTP-1-2-3 attack from reaching our routers / firewalls.  They surprised me when they said they didn't do that kind of work!


    More calls to escalate the filter request to the ISP eventually got to the right level, and the ISP's engineers agreed to filter inbound UDP/TCP ports 123 and things got better immediately.  It seems multiple clients of theirs may have also been experiencing the same reflection attack participation--NPM made us the first to contact the ISP and report the issue.


    It's tense when multiple levels of management are breathing down your neck and you've done all the right things.  When I later saw the Tech articles about that event I learned that multiple NTP-vulnerable sites had been chosen for reflection attacks against that same South American company, and the Distributed DOS attack successfully overwhelmed their big/multiple Internet pipes and firewalls for more than a day.


    Thankfully Orion Network Performance Monitor was the right tool for the job, the single tool I needed to aggregate the information into one place, enabling me to quickly diagnose and correct the problem.