14 Replies Latest reply on Nov 17, 2015 3:11 PM by sja

    Help... how can I alert on a sharp change in the WAN Utilization?

    sja

      Help SW

       

      I like to alert on a sharp change in my WAN Utilization.

       

      something like

      Interface Utilization is in 1 poll 30% and the next poll is 90% (+60% in 15 mim)

      Or from 3% to 60% (+57% change in 15 min)

       

       

      Why ?

       

      1.DDOS

      2.DDOS

      3.more DDOS

        • Re: Help... how can I alert on a sharp change in the WAN Utilization?
          deverts

          @sja,

           

          Have you looked at the canned alerts, "High RX Percent Utilization with Top Talkers" and "High TX Percent Utilization with Top Talkers"? These work great for my needs. One small drawback, since Netflow is exported every 5 minutes, I set the alert trigger to wait until 10 minutes of collection has passed, this way I get current conversations in the alerts.

           

          Dwyane

          • Re: Help... how can I alert on a sharp change in the WAN Utilization?
            rob.hock

            We implemented dynamic thresholds in 10.7, which calculate standard deviations on a 7-day sliding window. The dynamic value shows up as a macro, so you could potentially also use it elsewhere. Just go to edit an interface and you'll see the threshold override capability show up.

              • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                sja

                Sure will try that Rob.

                Is there any any materiel on that new feature?

                I enable that on my RC but the link to the materiel  is dead.

                 

                /SJA

                • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                  wluther

                  rob.hock This sounds very helpful, as we have been in this same boat for a while now.  Currently, we use "NFSEN" to monitor and alert us of network spikes (ddos), which seem to be happening on a more regular basis lately.  NFSEN is a very lightweight and extremely useful tool to easily monitor for these attacks.  We also have NTA, but find it more cumbersome to navigate and alert from.

                   

                  Are you saying this new feature will know, for example, if our uplinks are usually at/around 2gb at 5am, 6gb at 12pm, and 12gb at 8pm (with all the averages for all the times in between), that if there is 6gb traffic at 5am (network spike) it will alert from that BUT would NOT alert if it were 6gb at 12pm (normal)...?  As we have it now, I basically have to break it down between peaks and no traffic events.

                   

                  Also, not to hijack this thread, but what about linking/correlating endpoint IP address to bandwidth spike?  Currently, when we see a large spike in bandwidth, our NFSEN server sends us an email with the time, IP address, flow count, traffic size, duration, etc... Then starts our efforts to track down that user/endpoint, mitigate attack, and take various other actions... It would surely be nice to use our, paid for, NPM & NTA modules to do this, instead of a simple and lightweight free tool...

                   

                  Sounds like I need to schedule some maintenance time to upgrade NPM to 10.7 (currently at 10.6.1, NTA at 4.0.0) to use this new dynamic feature, as long as I am understanding what you are saying, that is...

                   

                  Thank you,

                   

                  -Will

                    • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                      rstoney00

                      The other way is to create an alert that compares the utilization over the last (for example) hour, vs the average utilization over the least 1 or 2 weeks.  From there, calculate your delta, and alert if the delta is above a certain percentage. 

                        • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                          sja

                          Hi  rstoney00 That  type of alert will sure help. Du you have alert example i can use? /sja

                            • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                              rstoney00

                              This would be part of a Custom SQL alert.   The top part is already given to you on the Interface Custom SQL, so you just need to use the parts from the inner join down.  It looks for a custom property on the interface of "ThresholdCheck"  so you can determine which circuits are involved.   This checks the utilization from 9 a.m. to 5 p.m. from the last 7 days, and compares it to the utilization of the last hour, and calculates the delta.   You probably do not want any shorter compare time frame for the hour, else you will get alerts from someone downloading an ISO, movie, torrent, sending a 125 mb attachment to their 100 best friends on the Internet, etc.

                               

                              The delta values at the bottom can be changed from 60, or to have different Rx / TX values.  I would guess if you wanted different values for different interfaces, you could just use multiple custom properties to sort that out.    Enjoy.

                               

                               

                               

                               

                              Select

                              Interfaces.InterfaceID as NetObjectID,

                              Interfaces.Fullname AS Name

                               

                              FROM Interfaces

                               

                              INNER JOIN

                               

                              (Select

                              T2.NodeName,

                              T2.InterfaceID,

                              T2.Delta_RX,

                              T2.Delta_TX

                               

                               

                              FROM

                              (Select

                              T1.NodeName,

                              T1.Caption,

                              T1.InterfaceID,

                              T1.TAvgRx,

                              W1.WAvgRx,

                              T1.TAvgTx,

                              W1.WAvgTx,

                              ROUND ((((T1.TAvgRx - W1.WAvgRx) / nullif(W1.WAvgRX,0))*100),0) AS Delta_Rx,

                              ROUND ((((T1.TAvgTX - W1.WAvgTx) / nullif(W1.WAvgTX,0))*100),0) AS Delta_Tx

                               

                                  From

                                          (Select

                                              H1.NodeName,

                                              H1.Caption,

                                              H1.InterfaceID,

                                              ROUND (AVG(H1.In_Averagebps),0)AS TAvgRx,

                                              ROUND (AVG(H1.Out_Averagebps),0)AS TAvgTx

                                            

                                              FROM  

                                                  (Select

                                                      N.Caption as NodeName,

                                                      I.Caption,

                                                      I.InterfaceID,

                                                      ITD2.In_Averagebps,

                                                      ITD2.Out_Averagebps

                               

                                                      From dbo.Nodes N Inner JOIN

                                                        dbo.Interfaces I on N.NodeID = I.NodeID Inner Join

                                                        dbo.InterfaceTraffic_Detail ITD2 on I.InterfaceID = ITD2.InterfaceID

                                                      WHERE I.ThresholdCheck = '1'

                                                      AND ITD2.DateTime Between DATEADD(minute, -60, GETDATE()) and GETDATE()

                                                              )H1

                                                      

                                              Group By H1.InterfaceID, H1.Caption,H1.NodeName

                                              )T1

                               

                                  INNER JOIN

                                            (Select

                                          WAvg.NodeName,

                                          WAvg.Caption,

                                          WAvg.InterfaceID,

                                          ROUND (AVG(WAvg.In_Averagebps),0)AS WAvgRx,

                                          ROUND (AVG(WAvg.Out_Averagebps),0)AS WAvgTx

                               

                                          FROM

                                                  (Select

                                                                  N.Caption as NodeName,

                                                                  I.Caption,

                                                                  I.InterfaceID,

                                                                  ITD.In_Averagebps,

                                                                  ITD.Out_Averagebps

                               

                                                                  From dbo.Nodes N Inner JOIN

                                                                    dbo.Interfaces I on N.NodeID = I.NodeID Inner Join

                                                                    dbo.InterfaceTraffic_Detail ITD on I.InterfaceID = ITD.InterfaceID

                                                                    

                                                                  WHERE I.ThresholdCheck = '1' and

                                                                  DateTime Between

                                                                  DateAdd(hh,9,(DATEADD(day, -7 ,DATEADD(wk, DATEDIFF(wk,0,GETDATE()), 0)))) AND

                                                                  DateAdd(hh,17,(DATEADD(day, -7 ,DATEADD(wk, DATEDIFF(wk,0,GETDATE()), 0))))

                               

                                                  )Wavg

                               

                                          Group by Wavg.NodeName, Wavg.Caption, Wavg.InterfaceID

                                              )W1

                                  ON T1.InterfaceID = W1.InterfaceID)T2 where Delta_Rx >= 60 OR Delta_Tx >= 60

                              )FT on FT.InterfaceID = Interfaces.InterfaceID

                                • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                                  pserwe

                                  How does this custom sql alert translate into 11.5.2 with the whole new web based alerting engine?

                                   

                                  Second question for Rob or anyone else on this thread, have we implemented any ToD/DoW/DoM functionality as of yet?


                                  Netflow is a non-starter, no netflow license, and honestly, from past experience, not really worth it to solve a traffic delta problem - i.e.:  I don't need to know precisely what traffic.

                                   

                                  My use case is that there was a major event where several, if not all interfaces in a given area dropped to 25% of normal traffic, and the NOC phones rang off the hook after about a 10-15 minute interval.  I'd like to see alerts pushed out before the phones ring.

                                   

                                  The 7 day moving average might work, I have to look into it / test it, but really interface traffic delta is what we really need, other than about 1-2 hours of ramp up / ramp down time, traffic is pretty consistent business hours.

                                   

                                  This would be in the category of "major network outage alerts".. aka stuff you don't want to just see in the graphs.

                                   

                                  Peter

                            • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                              rob.hock

                              The new dynamic baseline does not take into account time-of-day/day-of-week/etc, but rather is a 7-day moving average with calculated standard deviations. The ToD/DoW/DoM functionality would be ideal, but is not present in 10.7

                          • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                            noman4ever

                            Is there any other software that can yield this functionality?

                              • Re: Help... how can I alert on a sharp change in the WAN Utilization?
                                wluther

                                noman4ever We use NFSEN for this purpose. It is free, lightweight, and fairly easy to use. It is a very simple solution, and will alert us very quickly.  Unfortunately, we have not seen this basic functionality implemented within NTA/NPM.

                                 

                                Here are the subject lines from the emails we get on a few of our NFSEN alerts:

                                 

                                • Uplink1 link traffic has dropped 1mbps below 10min average or is below 2kbps total - check NFSEN status now!
                                • Uplink2 link traffic has dropped 1mbps below 10min average or is below 2kbps total - check NFSEN status now!
                                • Greater than 5k UDP flow dDOS attack detected to one IP address - check NFSEN status now!
                                • Greater than 10k flows detected to or from one IP address - check NFSEN status now!

                                 

                                It would be a welcomed addition for SolarWinds to incorporate the same functionality that NFSEN uses, as well as MRTG/Cacti for graphing. (Which have been asked for by many users of the years...)

                                 

                                If NTA would work as efficiently and effectively as NFSEN, I would shut down our NFSEN server and go 100% NPM/NTA. However, as is, I would have to recommend NFSEN to best do the job you are requesting.

                                 

                                -Will

                                1 of 1 people found this helpful