15 Replies Latest reply on May 25, 2016 3:06 PM by xr219

    Discrepencey in NTA data and billed data

    xr219

      Hi all,

       

      We use NTA to count the traffic flowing to and from our customer who has a fixed IP address. When we look at the data in NTA so far this month the customer appears to have used a total of 27Mb (in and out). However, the CDR from our network supplier (CDR for those who aren't aware is a bit like our wholesale bill from our supplier) it shows 90 or so Mb. We cannot find the missing traffic. The flows are coming from the Ethernet interface on our router (Netflow v9) and this is the only way our end customer can get to anything, i.e. all of their traffic must traverse this interface.

       

      We have run the same period for other customers and the data is within 1mb of the CDR in every case.

       

      Is there any data that Netflow or NTA cant count? Are their limitations that I'm not aware of? Can it count traffic that is contained within a VPN etc?

       

      Excuse if this is a stupid question, but I would very much appreciate if those with more knowledge than me could explain it.

       

      many thanks.

        • Re: Discrepencey in NTA data and billed data
          xr219

          Hi guys,

           

          Sorry to bounce this, but does anyone have any idea why the Netflow Data volumes could be significantly (60 - 70 %) lower that the counted data using Wireshark?

          To try and get some clarity we put a port mirror on the switch port for Wireshark to Analyse the data. in a 30 minute period, Wireshark was showing 142KB... Netflow was showing 52KB. Is there some fundamental limitation of Netflow that we are missing. Any help would be appreciated.

            • Re: Discrepencey in NTA data and billed data
              rharland2012

              Is there a difference in the type of data used by this particular customer? Lots of UDP, for example? Or perhaps the particular data flows are not one you currently monitor?

                • Re: Discrepencey in NTA data and billed data
                  xr219

                  Hi Rharland2012,

                   

                  Sorry for the delay in getting back to you. We've made some progress (we think.) From the Wireshark, it looks like it may be DNS requests. I'm not sure exactly how Netflow / NTA handles such small requests, but when we look at the traffic between the endpoint and the internal DNS server over a 2 hour period, NTA is showing 27kb. Wireshark for the same period is showing approx. 470kb. We are working on the theory that because each DNS request is so small, they either aren't being counted by Netflow, or NTA isn't showing them in whatever way it counts the data. When we extrapolate the 2 hour data count over 1 month, it equates to about 170Mb. Does this sound possible? If so, is there a way to show these small data transactions, without adversely affecting performance?

                    • Re: Discrepencey in NTA data and billed data
                      rharland2012

                      In my experience, I haven't seen Netflow not record a conversation due to size. If Netflow sees the packet go in or out of the interface and can attach an interface index to the egress or ingress, it gets measured and added to a netflow datagram.

                      I'm assuming you've verified in your Netflow settings that DNS is listed under the 'Enabled Applications' tab. If not, I'd check that.

                      I'm also assuming that NONE of this DNS traffic shows up in your Netflow output. Is this accurate?

                        • Re: Discrepencey in NTA data and billed data
                          xr219

                          Hi,

                           

                          Thanks for getting back to me. We are seeing some DNS in the same time window, but much less than in Wireshark. We're using the default V9 Netflow setting from the router and running a conversation report showing traffic in both directions over the same time period. Could it be that NTA is summarizing this data in some way? Is there something I could share on here that may help shed light on this? 202dns.JPG

                            • Re: Discrepencey in NTA data and billed data
                              rharland2012

                              See this thread:

                               

                              Netflow Nexus 7k showing up with inaccurate utilization!

                               

                              Check a) what sampling rate is set to and b) if any overrides are currently in place. If auto-detect didn't grab it, you may need to manually set the sample rate.

                              1 of 1 people found this helpful
                                • Re: Discrepencey in NTA data and billed data
                                  xr219

                                  Thanks for that. We are seeing AutoDetect:No Sampling. So I've changed this to Override Autodect and set it to 1 of 1 flows. Excuse my ignorance, but does that mean that the flow data is sent to NTA and by default it samples the flow data rather than reading everything in it? That's why I've chosen 1 of 1. I clearly have not got a good understanding of how the flow data is interpreted.

                                    • Re: Discrepencey in NTA data and billed data
                                      rharland2012

                                      If auto saw no sampling - and that is the accurate configuration on the router port - then you don't have to do anything.

                                      From the router's perspective, sampling simply means that it looks at 1/x  of the packets to extrapolate Netflow information. This is usually done to ease CPU pressure on the router or device originating the flows.

                                      Can you share a redacted/scrubbed Netflow config from your router so I can see what the router is sending to your collector?

                                        • Re: Discrepencey in NTA data and billed data
                                          xr219

                                          Hi, Thanks for that, it makes sense.

                                           

                                          My Network engineer has provided the follow. Its a v9 flow, but with no customization, other than to send the flows to the poller.

                                           

                                          interface GigabitEthernet0/0
                                          description Connection to NLL3S002 (172.20.20.126)
                                          ip address 10.50.40.98 255.255.255.248
                                          ip flow ingress
                                          ip flow egress
                                          duplex auto
                                          speed auto

                                           

                                          ip flow-export source GigabitEthernet0/0
                                          ip flow-export version 9
                                          ip flow-export destination 10.8.93.81 2055

                                            • Re: Discrepencey in NTA data and billed data
                                              rharland2012

                                              Okay - looks good.

                                              There should also be some global Netflow commands on the router that I would verify if I was troubleshooting this on my own network.

                                              Can you verify with your network engineer what the settings are for these?

                                               

                                              ip flow-cache timeout active

                                              ip flow-cache timeout inactive

                                               

                                               

                                                • Re: Discrepencey in NTA data and billed data
                                                  xr219

                                                  Hi,

                                                   

                                                  Just spoken with him and the values aren't set so they will be the router defaults. The device is a Cisco 2921.

                                                   

                                                  We're only using flows on this device as its the de-mark connection to our customers, so all traffic flows through it. When a customer queries their bill, we can run a Netflow report for the period that gives some degree of detail for them. The issue we have is that, although small, this constant DNS traffic can account for 100 - 200 Mb per month, which can equate to a lot of money. If we cant see this usage in the Netflow reports, we have a significant discrepancy in the bill and the reports that we can provide them, and so they are reluctant to pay. That's why we are really keen to get a detailed view of the usage, and it should marry up almost exactly (within 10%) of the billed volume of data. We don't need to run flows on lots of devices.  

                                                • Re: Discrepencey in NTA data and billed data
                                                  Craig Norborg

                                                  I tend to do "ip flow ingress" on all the ports rather than do both on one port.   Tends to be more accurate, esp. if you put both on multiple ports.   However, if that was the problem you'd be seeing more Netflow data rather than less.   It would double the traffic on you.

                                                   

                                                  Guessing you might have these settings done already, but didn't see it above...  In your NTA settings in Orion.   Near the top do you have it checked to "Enable data retention for traffic on unmonitored ports"?    That can make a huge difference..    I'd also make sure you check "Allow monitoring of flows from unmanaged interfaces" and "Allow matching nodes by another IP Address". 

                                                   

                                                  Also go into "Applications and Service Ports" and "Monitored Protocols" and make sure everything is checked or enabled.    ie: "Enable All Monitoring" in the first.   You can also add any applications, service ports or protocols that you might know you have but aren't in here...

                                                   

                                                  Back up near the top you can also "Show unknown traffic events" and see if anything might be in there that might help...

                                                   

                                                  Also, what kind of box is it?  A Cat6K for example you would need to make sure you did "mls netflow" in order to get the switched traffic from the PFC.  Knowing the hardware you're on might help us figure it out.

                                                  1 of 1 people found this helpful
                                                    • Re: Discrepencey in NTA data and billed data
                                                      xr219

                                                      The device is a Cisco 2921.

                                                       

                                                      We're only using flows on this device as its the de-mark connection to our customers, so all traffic flows through it. When a customer queries their bill, we can run a Netflow report for the period that gives some degree of detail for them. The issue we have is that, although small, this constant DNS traffic can account for 100 - 200 Mb per month, which can equate to a lot of money. If we cant see this usage in the Netflow reports, we have a significant discrepancy in the bill and the reports that we can provide them, and so they are reluctant to pay. That's why we are really keen to get a detailed view of the usage, and it should marry up almost exactly (within 10%) of the billed volume of data. We don't need to run flows on lots of devices.  

                                  • Re: Discrepencey in NTA data and billed data
                                    xr219

                                    Hi Guys,

                                     

                                    having gone back over the NTA settings (thanks for the prompt Chris) It looks like the error was down to Top Talker optimisation which was at the default 95%. For now we have pushed this up to 100% and will run in this way for 24 hrs to get some data, but comparing the first hours worth of data, it looks much better. It may be that we can tweak it down one or two percentage points, but we will need to do this after we have some meaningful data at 100% before we can compare results.

                                     

                                    Thanks so much for your input. I'll update this thread once we have data at 100%, 99%, 98% and 97% and let you know what I find. It may be helpful to others in the future.