15 Replies Latest reply on Apr 17, 2018 4:24 PM by rschroeder

    Unable to capture NetFlow on Cisco 3750x switch

    WinskiTech711

      A few years back I had SolarWinds Real-Time NetFlow Analyzer working with my Cisco 3750x switch. Recently we had some suspicious traffic so I installed a new version of the software on a Win 10 machine. I can connect through the software to my switch, I can see all of the interfaces but none of them show NetFlow enabled. When I click on the interface I want to monitor then click "Start Flow Capture" I get a 'NetFlow is not detected on the selected interface'.

       

      How do I get this port configured correctly to capture NetFlow data?

       

      Additional Facts:

      IOS version 15.0(2)SE6

       

      Config:

      int gig <port to be monitored>
      ip flow ingress
      ip flow egress

      ip flow-export source <port to be monitored>
      ip flow-export version 5
      ip flow-export destination <IP of my Win 10 machine> 2055

       

       

      Per this thread- https://thwack.solarwinds.com/thread/20498

      I tried to run the ip nbar protocol-discovery and the ip route-cache flow on the port to be monitored. Neither of those commands were accepted on that port.

       

      Any help is appreciated.

        • Re: Unable to capture NetFlow on Cisco 3750x switch
          rschroeder

          I've been able to use Solarwinds' documentation for getting Netflow configured on a good variety of Cisco devices.  3750's, Nexus 7K's and 5K's, 4510's, and a bunch of routers.  I use this basic guideline, and tweak and tune the commands based on individual platform limitations or requirements, which can be found if you Google Netflow and that particular Cisco box.

           

           

           

           

          Set up NetFlow NBAR2 on Cisco devices

          Network Based Application Recognition (NBAR) is the mechanism used by certain Cisco routers and switches to recognize a dataflow by inspecting some of the packets sent. SolarWinds NTA 4.2.1 supports unknown traffic detection and advanced application recognition through NBAR2.

          First, configure your Cisco devices to send NBAR2 data to SolarWinds NTA. Second, add those devices as nodes in SolarWinds NPM and SolarWinds NTA.

          The following values are examples used in the commands below:

          • NTArec

           

          • NTAexp
          • NTAmon
          • GigabitEthernet0/1
          • 10.10.10.10

          Create a new Flexible NetFlow configuration

           

          Add the flow record

          This process is similar to creating a standard NetFlow configuration. In this case, you add the collect application name command to enable the sending of AppID in each flow.

          flow record NTArec

          match ipv4 tos

          match ipv4 protocol

          match ipv4 source address

          match ipv4 destination address

          match transport source-port

          match transport destination-port

          match interface input

          collect interface output

          collect counter bytes

          collect counter packets

          collect application name

          exit

          Add the flow exporter

          The option application-table command enables the sending of a list of applications that can be classified using NBAR2, including applications that were manually created. The option application-attributes command enables the sending of categories for all applications.

          flow exporter NTAexp

          destination 10.10.10.10

          source GigabitEthernet0/1

          transport udp 2055

          export-protocol netflow-v9

          template data timeout 60

          option application-table timeout 60

          option application-attributes timeout 300

          exit

          Add the flow monitor

          The flow monitor connects the flow recorder and the flow exporter. You can configure multiple recorders, exporters, and monitors at once.

          flow monitor NTAmon

          description NetFlow nbar

          record NTArec

          exporter NTAexp

          cache timeout inactive 30

          cache timeout active 60

          exit

          When receiving long flows, these values may need to be adjusted, see Troubleshoot Long Flow Errors for more details. For more information about the timeout values, refer to the Cisco NetFlow Command Reference.

          Apply the monitor on an interface

          Assign the Flexible NetFlow configuration to the interface from which to monitor NetFlow.

          interface GigabitEthernet0/1

          ip flow monitor NTAmon input

          ip flow monitor NTAmon output

          exit

          Diagnostic commands

          show flow record "recordName"

          show flow export "exporterName"

          show flow monitor "monitorName"

          show flow exporter statistics

          show flow interface

          Determine the applications your device can recognize

          The Protocol Pack is a list of applications, definitions, and categories that your device can recognize.

          Check the Protocol Pack version

          show ip nbar version

          View a list of the available applications

          show ip nbar protocol-id

          Edit an existing record

          If you edit an existing record that is in use, you receive the following error:

          % Flow Record: Flow Record is in use. Remove from all clients before editing.

          To resolve this error, remove the connection between the monitor, record, and interface.

          Disable the connection

          interface GigabitEthernet0/1

          no ip flow monitor NTAmon input

          no ip flow monitor NTAmon output

          exit

          Add the application recognition field into the record

          flow record NTArec

          collect application name

          exit

          Add the application recognition field into the exporter

          flow exporter NTAexp

          option application-table timeout 60

          option application-attributes timeout 300

          Restore the connection

          interface GigabitEthernet0/1

          ip flow monitor NTAmon input

          ip flow monitor NTAmon output

          exit

           

           

           

           

           

           

           

          So let's say you have a Cisco 4510.  Here's my copy-and-paste instructions, minus the unique IP addresses or interfaces you need to add:

           

           

           

           

          How To Set Up Netflow on Cisco 4510 Version 8 Chasses:

           

          1. The switch hardware must be Version 8 or newer.  V7 and older requires NetFlow Modules to be purchased and installed in each Supervisor.
          1. The chassis must be licensed to run IP Base or Enterprise.  NetFlow is not supported on LAN Base license.

           

           

          conf t

           

          flow record NTArecord

          match ipv4 tos

          match ipv4 protocol

          match ipv4 source address

          match ipv4 destination address

          match transport source-port

          match transport destination-port

          match interface input

          collect interface output

          collect counter bytes

          collect counter packets

          collect timestamp sys-uptime first

          collect timestamp sys-uptime last

          !

          flow exporter NTAexport

          destination x.x.x.x (You add in your SW Poller's address here)

          source Loopback0 (Or use a different interface--whatever you use to manage the switch is the interface to report with)

          transport udp 2055

          export-protocol netflow-v5

          !

          flow monitor NTAmonitor

          description NetflowToOrion

          exporter NTAexport

          cache timeout inactive 10

          cache timeout active 5

          record NTArecord

           

          Add “ip flow monitor NTAmonitor input“ to every VLAN you want included.  You can also group them via this example:

           

          vlan configuration (Insert ALL the VLAN's on the 4510 in this area)

          ip flow monitor NTAmonitor input

           

          On the WAN interface's physical port(s):

          ip flow monitor NTAmonitor input

           

          Add this line for EVERY physical port to want to monitor on the switch: 

          ip flow monitor NTAmonitor input

           

           

          ! Modify the interface script that follows based on the modules you own:

          conf t

          int range gi1/1-48,gi2/1-48,gi3/1-48,gi4/1-48

          ip flow monitor NTAmonitor input

          int range gi7/1-48,gi8/1-48,gi9/1-48,gi10/1-48

          ip flow monitor NTAmonitor input

           

          Then tell the switch which interface to use as its Netflow source.   A 4510 serving as a WAN router and Distribution switch should use a loopback port, but you could choose the physical WAN interface.  Use the same port as is used by the switch for all its sourcing of logging, TACACS, snmp, etc.

           

          Build the exporter, then assign it to the correct Interface so Orion doesn’t throw a bunch of errors about an unmanaged device sending it Netflow info.

           

          Example: 

           

          conf t

          flow exporter NTAexport

          description LSEG internal

          destination (x.x.x.x is the IP address of your Solarwinds Poller)

          source Loopback0

          transport udp 2055

          export-protocol netflow-v5

           

           

          int loopback0

          flow monitor NTAmonitor

          exporter NTAexport

          record NTArecord

           

          Finally, ensure NPM is set to monitor all interfaces that have the “ip flow monitor NTAmonitor input“ command.  If it’s not, then it’ll send NTA interface errors.

           

           

           

          Removal is the reverse of the steps above, in this order:

           

          int loopback1

          no flow monitor FLOW-MONITOR-1

          no exporter EXPORTER-1

          no record NTArecord

           

          no flow exporter EXPORTER-1

           

          int range gi1/1-48,gi2/1-48,gi3/1-48,gi4/1-48

          no ip flow monitor NTAmonitor input

          int range gi7/1-48,gi8/1-48,gi9/1-48,gi10/1-48

          no ip flow monitor NTAmonitor input

           

          int range te5/1-8,te6/1-8

          no ip flow monitor NTAmonitor input

           

          vlan configuration x-x

          no ip flow monitor NTAmonitor input

           

          no flow monitor NTAmonitor

           

          no flow exporter NTAexport

           

          no flow record NTArecord

           

           

           

          Now let's suppose you had to do this on a 6509 Core or Distribution L3 switch.  Here's how:

           

          Enabling Netflow on 6509 Distribution Switches

           

           

          ip flow-cache entries 131072 (if you change this, the switch must be 

          rebooted or all flow must be removed before it takes effect)

           

          ip flow-cache timeout active 1

          ip flow ingress layer2-switched vlan x (must be done for every vlan)

          mls flow ip interface-full

          no mls flow ipv6

          mls nde sender version 5

           

          **VLAN/physical interface's**

          !  int vlan 2 (etc.  must be done for every SVI)

          ip flow ingress

          ip route-cache flow

           

           

          ip flow-export source lo0

          ip flow-export version 5

          ip flow-export destination x.x.x.x (this is the address of your Solarwinds server)  2055

           

           

           

          Let's say you want your ASA to report Netflow.  It's super easy:

           

          flow-export destination  ABCD  (the name of the ASA Interface that you want to send the Netflow traffic through--it might be really intuitive like "inside")   x.x.x.x (the IP address of your Solarwinds poller) 2055

           

           

           

          So you have 3750X's.  Are they compatible with NetFlow?

           

          If they ARE compatible, I recommend you use Solarwinds' Netflow configuration guidance.  But you can also refer to Cisco's info here:

          Catalyst 3750-X and 3560-X Software Configuration Guide, Release 15.0(1)SE - Configuring Flexible NetFlow [Cisco Catalys…

           

           

          Good Luck!  Let us know how it works out for you!  Send pictures--or it didn't happen!

           

          2 of 2 people found this helpful
          • Re: Unable to capture NetFlow on Cisco 3750x switch
            WinskiTech711

            rschroeder, I'm trying to analyze/capture netflow from the gigabit ports on my 3750x. That don't support flexible netflow. I've had non-flexible netflow working on my gigabit ports at some point in the past.

             

            Does the netflow analyzer only work with flexible netflow (one has to apply "ip flow monitor <name of flow monitor> input" on the specific interface they want monitored) now? The only netflow commands I can apply directly to the interfaces I want analyze are "ip flow ingress" and "ip flow egress".

              • Re: Unable to capture NetFlow on Cisco 3750x switch
                rschroeder

                NTA supports both version 5 and 9, but I recommend using version 9 with NBAR2 everywhere you can.  Some legacy devices aren't compatible with NBAR2, others can only do Netflow v5.  Find which ones have that limitation and compensate for them, and request budget to replace them with newer models that support Netflow v9 and NBAR2.

                 

                I apply flow commands to every physical interface on my Cisco 4510 chasses now that the V8 model supports the commands, and it opens up another layer of granularity for traffic on a per-port basis.   In that particular environment, it's only possible to use the "ip flow monitor <name> input" command.  Initially I thought this was a limitation because there was no matching "output" command for the port.  It turns out that, while having both commands on the port seem intuitive and convenient, I'm really only interested in traffic coming "from" the device directly attached to the port.  Any traffic going "to" that device from another device is captured on the port(s) allowing the traffic into the switch from the other device.

                 

                Regarding your 3750x, getting its Netflow going again most likely will require a review of the required commands and a fine-toothed comb going through the details.  Although you had it working previously, since it's not working now, you may benefit from thinking about what's changed that caused it to stop.

                • Was there an IOS update or downgrade that resulted in different capabilities, or that needs different commands applied to get Netflow going again?
                • Did a destination address change for the Netflow?  If you updated/changed a Solarwinds Poller, it could still be polling the 3750x, but the 3750x might not be sending Netflow to the correct destination address.

                If you have NCM, I'd recommend comparing a running-configuration from the 3750x at the time it was properly sending Netflow to today's running-config.  Maybe you'll see a change or a typo.  Or perhaps you'll find something that SHOULD have changed, but hasn't, to support a different destination address for a Solarwinds poller.

                 

                I looked for a 3750x in my network that was running the right code and license level to use Netflow and I find I've retired them all.

                 

                 

                But here's a snip from one of my 4510's running Netflow on all interfaces that can be compared to your output:

                 

                flow record NTArec

                match ipv4 tos

                match ipv4 protocol

                match ipv4 source address

                match ipv4 destination address

                match transport source-port

                match transport destination-port

                match interface input

                collect interface output

                collect counter bytes

                collect counter packets

                collect application name

                flow exporter NTAexp

                destination <x.x.x.x> (your Solarwinds APE running NTA)

                source <enter the interface on the switch that will be recognized as the source of the traffic.  Always use the same Interface that is being polled by Solarwinds--usually an SVI or a loopback>

                transport udp 2055

                template data timeout 60

                option application-table timeout 60

                flow monitor NTAmon

                description NetFlow nbar

                exporter NTAexp

                cache timeout inactive 30

                cache timeout active 10

                record NTArec

                ip flow monitor NTAmon input (this command goes on every physical port)

                 

                vlan configuration (list all VLAN ID's here, comma-separated)

                  ip flow monitor NTAmon input

                 

                 

                You may have to tweak this a bit for your 3750x's, but it should get you very close to running again.

                 

                Swift packets!

                 

                Rick Schroeder

                  • Re: Unable to capture NetFlow on Cisco 3750x switch
                    WinskiTech711

                    This is the config I had/have on the switch, updated with the interface I want to monitor and the source ip of the new Netflow Analyzer. I have a feeling I'm getting tripped up on the ip flow-export source line, documentation did not make this clear. Right now I have it set as the interface I want monitored.

                     

                    flow record <record name>

                    match ipv4 tos

                    match ipv4 protocol

                    match ipv4 source address

                    match ipv4 destination address

                    match transport source-port

                    match transport destination-port

                    collect counter bytes

                    collect counter packets

                     

                    flow exporter <exporter name>

                    destination <netflow analyzer IP>

                    transport udp 2055

                     

                    flow monitor <monitor name>

                    description Original Netflow captures

                    record ipv4

                    exporter <exporter name>

                     

                    interface <interface to be monitored>

                    ip flow ingress

                    ip flow egress

                     

                    ip flow-export source <interface still not clear exactly this is for>

                    ip flow-export version 5

                    ip flow-export destination <netflow analyzer address> 2055

                    ip flow-top-talkers

                    top 10

                    sort-by bytes

                      • Re: Unable to capture NetFlow on Cisco 3750x switch
                        rschroeder

                        The "ip flow-export source" line tells the 3750x what IP address it should include as the "from" or "sender", when sending to your Solarwinds NTA poller.   For example, if you only have one IP address on the switch, and it's loopback0, then you'd say "ip flow-export source loopback0" on this line.

                         

                        If your 3750x has multiple IP addresses, always use the interface with the IP address that's being monitored by Network Traffic Analyzer for the "ip flow-export source".  It helps Solarwinds NPM and NTA keep everything aligned nicely when you use the same monitoring address that NPM knows about, for the source interface in NTA.

                         

                        If you don't do this, you'll be monitoring your switch in NPM with one IP address, and the switch will be sending Netflow information to NTA from an interface with a different IP address than the one NPM already is monitoring.  This will create an alert, and you'll be recommended to either add the new Netflow-associated IP address as an entirely new node (wasting license count and server resources), or you can simply change the "ip flow-export source" line to reference the Interface with an IP address that NPM monitors.

                          • Re: Unable to capture NetFlow on Cisco 3750x switch
                            WinskiTech711

                            rschroeder, that makes more sense than what I read. When I ran the ip flow-export source command I couldn't put in an address specifically, I had to put in an interface, so I used the interface I ssh into the switch on. That didn't work unfortunately, when I open up netflow analyzer I still get no flow type next to any of the interfaces.

                             

                            I've also been working on getting the netflow configurator working. When I try to connect to my device using my read only SNMP community string, the software says I need a read/write community string to continue. I created a read/write SNMP community string, tried that in the software, it says cannot connect to device. Not sure what the issue is there either.

                            • Re: Unable to capture NetFlow on Cisco 3750x switch
                              WinskiTech711

                              sh flow exporter NTAexp command results:

                               

                              Flow Exporter NTAexp:

                                Description:              User defined

                                Export protocol:          NetFlow Version 9

                                Transport Configuration:

                                  Destination IP address: <netflow collector>

                                  Source IP address:      <switch IP used to SSH in>

                                  Source Interface:       <above IPs interface>

                                  Transport Protocol:     UDP

                                  Destination Port:       2055

                                  Source Port:            56488

                                  DSCP:                   0x0

                                  TTL:                    255

                                  Output Features:        Not Used

                              • Re: Unable to capture NetFlow on Cisco 3750x switch
                                WinskiTech711

                                show flow interface command results:

                                 

                                Interface <interface used to SSH into switch>

                                  FNF:  monitor:          NTAmon

                                        direction:        Input

                                        traffic(ip):      on

                                  FNF:  monitor:          NTAmon

                                        direction:        Output

                                        traffic(ip):      on

                        • Re: Unable to capture NetFlow on Cisco 3750x switch
                          WinskiTech711

                          Below is the code I had on my switch when netflow analyzer was working:

                           

                           

                          flow record <record name>

                          match ipv4 tos

                          match ipv4 protocol

                          match ipv4 source address

                          match ipv4 destination address

                          match transport source-port

                          match transport destination-port

                          collect counter bytes

                          collect counter packets

                           

                          flow exporter <exporter name>

                          destination 10.1.1.25

                          transport udp 2055

                           

                          flow monitor <monitor name>

                          description Original Netflow captures

                          record ipv4

                          exporter <exporter name>

                           

                          interface <interface to be monitored>

                          ip flow ingress

                          ip flow egress

                           

                          ip flow-export source <interface still not clear exactly this is for>

                          ip flow-export version 5

                          ip flow-export destination <netflow analyzer address> 2055

                          ip flow-top-talkers

                          top 10

                          sort-by bytes

                          • Re: Unable to capture NetFlow on Cisco 3750x switch
                            rschroeder

                            It seems our hardware types and IOS versions aren't close enough to be completely compatible with each other, and my examples may not be helpful.

                             

                            It may be time to open a TAC case AND a ticket with Solarwinds Support, to ensure you have

                            both the correct and compatible Cisco IOS/Hardware/Commands/Licenses/Capabilities on

                            the switches, and the right expectations on the Solarwinds side.

                             

                             

                            Here are the "show flow" outputs for one of my 4510's that is working well with NTA:

                             

                             

                             

                             

                            #sho flow exporter

                            Flow Exporter NTAexp:

                              Description:              User defined

                              Export protocol:          NetFlow Version 9

                              Transport Configuration:

                                Destination IP address: <Solarwinds APE IP address>

                                Source IP address:      x.x.x.x (the SVI of the Management VLAN on the switch)

                                Source Interface:       (the VLAN hosting the IP address)

                                Transport Protocol:     UDP

                                Destination Port:       2055

                                Source Port:            60156

                                DSCP:                   0x0

                                TTL:                    255

                                Output Features:        Not Used

                              Options Configuration:

                             

                             

                             

                             

                            #show flow interface

                            Interface GigabitEthernet1/1

                              FNF:  monitor:          NTAmon

                                    direction:        Input

                                    traffic(ip):      on

                            Interface GigabitEthernet1/2

                              FNF:  monitor:          NTAmon

                                    direction:        Input

                                    traffic(ip):      on

                            (the above extends to all 384 physical ports on the chassis switch)

                             

                             

                             

                             

                             

                             

                            #show flow monitor

                            Flow Monitor NTAmon:

                              Description:       NetFlow nbar

                              Flow Record:       NTArec

                              Flow Exporter:     NTAexp

                              Cache:

                                Type:                 normal

                                Status:               allocated

                                Size:                 4096 entries / 278544 bytes

                                Inactive Timeout:     30 secs

                                Active Timeout:       10 secs

                                Update Timeout:       1800 secs

                                Synchronized Timeout: 600 secs

                             

                             

                             

                             

                             

                             

                            #sho flow record

                            flow record NTArec:

                              Description:        User defined

                              No. of users:       1

                              Total field space:  34 bytes

                              Fields:

                                match ipv4 tos

                                match ipv4 protocol

                                match ipv4 source address

                                match ipv4 destination address

                                match transport source-port

                                match transport destination-port

                                match interface input

                                collect interface output

                                collect counter bytes

                                collect counter packets

                                collect application name