23 Replies Latest reply on Mar 30, 2010 10:06 AM by jalexander

    Giving up on NTA - Last Ditch Efforts

      Hello there, here is an outline of my situation:

      1.  I am attempting to collect NetFlow on a Cisco 2811 router and a Cisco 2821 router.  The routers and the SolarWinds server/collector (Server 2008) are on the same subnet.

      a.  There are ACL's specifically in place on the routers to allow for NetFlow to reach the collector.  These have been examined by Support.

      b.  There are firewall exceptions in place on the SolarWinds server/collector to allow for all necessary UDP and TCP ports.  These were added by Support.

      2.  I have spent well over 20 hours with support on the phone, through e-mail, and in GoToMeetings.  We have exhausted the possibility of SolarWinds causing the issue.

      a.  SolarWinds recognizes NetFlow on UDP 2055 when generated from other sources (using a generic generator).

      b.  Traffic In, Traffic Out, and Last NetFlow Received all display data for GigabitEthernet 0/0, 0/1 (Cisco 2821) and FastEthernet 0/0, 0/1 (Cisco 2811).

      3.  WireShark shows incoming NetFlow on UDP 2055 when generated from other sources.  WireShark shows no incoming NetFlow on UDP 2055 when generated from the 2811/2821.

      4.  I have used the NetFlow Configurator utility.  I have used the newly-released Config Generator w/NetFlow template to generate the appropriate script (which was consequently uploaded via NCM).

      5.  Using show ip flow export lists no failures for NetFlow v5, but the flows never reach the collector.

      If there is something here that I am missing, or if there is someone in the community that could help me step through the process of enabling NetFlow correctly on a 2811/2821, please help.  I have reached the end of my troubleshooting skills for the third time now and would still like to have this running/configured as soon as I possibly can.

      Thank you in advance for any help or suggestions,

      Jeremy

        • Re: Giving up on NTA - Last Ditch Efforts
          kweise

          Jeremy,

          There isn't a lot to the NetFlow config on the 2800s. 

          ip flow-export source <interface> - the key here is you have to source NetFlow from the same interface that has the IP address of the node in Orion.  For example, if the node was added to Orion using the IP address on the serial interface, you have to set the ip flow-export source to that serial interface.

          ip flow-export version 5

          ip flow-export destination <ip address of your Orion server> 2055

           

          ip flow ingress or ip flow egress on your interface config(s)

           

          Hope this helps.  If not, let us know, maybe there is something else we can take a look at.

            • Re: Giving up on NTA - Last Ditch Efforts
              njoylif

              to piggy back "kweise", we use Loopbacks for orion to manage nodes, so we have to source netflow from loopback.

              secondly, wondered if you tried sniffing by the router to see if it was outputting netflow..could be bug in IOS maybe?
              maybe do packet capture on each interface side of the firewall to see if coming in and going out through firewall...

              good luck

              • Re: Giving up on NTA - Last Ditch Efforts

                Hi Jeremy--

                Sorry you are having trouble. I'm going to route this over to the NTA product manager and development as well to see if they can help.

                M

                  • Re: Giving up on NTA - Last Ditch Efforts

                    First, I have never posted to thwack before, and I am quite impressed by the speed and number of replies.  Regardless of solving the problem, this is a very comforting thing.

                    Secondly, as an example I'll use the Cisco 2811.

                    The IP Address of the Cisco 2811 router is 172.16.100.247.  The address on the 0/0 interface is 172.16.100.247.  I am using this interface as the NetFlow source.  Ingress and Egress have both been enabled on 0/0 and 0/1.  Is this correct?  Only the 0/0 interface is relevant, as both of these routers move traffic from outside the organization to inside (predominantly Internet-activities).

                    Fianlly, while I would gladly try out what njoylif has to say, I'm afraid I'm a lowly programmer and not that familiar with the tools available to me in evaluating this situation.  I do understand a lot of the basics, but would appreciate any additional information on how to accomplish the two tasks you suggested.

                    I would also like to thank MarieB for the forward.

                      • Re: Giving up on NTA - Last Ditch Efforts
                        njoylif

                        on the (cisco) firewall, you should be able to do what is called a capture.  You will have to define an interesting traffic ACL and apply that to the targeted interface.  You can run multiple on the device, one per interface.

                        your interesting traffic can be very specific or very general...since you know the specifics - it might look something like this...
                              access-list dmzside permit udp host 172.16.100.247 host x.x.x.x (Netflow server IP) eq 2055

                        **** this assumes you are familiar with NAT and whether or not you guys are doing that...

                        you would apply that to the interface facing the router by doing something like:
                             capture <capname> access-list dmzside interface <interface nameif>

                        This will not affect production traffic <cya> assuming there is not a TON of data to flood the link and it is configured correctly</cya>

                        you would repeat steps for inside interface (side to get to netflow system).  DO NOT use the same ACL.  even if same, create another and use different name.

                        good luck.

                          • Re: Giving up on NTA - Last Ditch Efforts
                            pyro13g

                            Add Netflow to at least one other interface on each router and see what happens.

                            • Re: Giving up on NTA - Last Ditch Efforts
                              jswan

                              njoylif, it sounds like there are no ASAs in the path of the traffic, so I don't think he's going to be able to set up an ASA capture as you describe.

                              Jeremy, can you post a sanitized version of your entire router config? By sanitized, I mean remove all passwords and public IP addresses?

                                • Re: Giving up on NTA - Last Ditch Efforts

                                  jswan, you are correct in that there are no ASAs in the path of traffic.  As requested, here is the current running config on my Cisco 2811 router:

                                  001: !
                                  002: version 12.4
                                  003: no service pad
                                  004: service tcp-keepalives-in
                                  005: service tcp-keepalives-out
                                  006: service timestamps debug datetime msec localtime show-timezone
                                  007: service timestamps log datetime msec localtime show-timezone
                                  008: service password-encryption
                                  009: service sequence-numbers
                                  010: !
                                  011: hostname <hostname>
                                  012: !
                                  013: boot-start-marker
                                  014: boot-end-marker
                                  015: !
                                  016: security authentication failure rate 3 log
                                  017: security passwords min-length 6
                                  018: logging buffered 4096 debugging
                                  019: enable secret 5 <secrets>
                                  020: !
                                  021: aaa new-model
                                  022: !
                                  023: !
                                  024: aaa authentication login local_authen local
                                  025: aaa authorization exec local_author local
                                  026: !
                                  027: aaa session-id common
                                  028: !
                                  029: resource policy
                                  030: !
                                  031: clock timezone UTC -6
                                  032: clock summer-time UTC recurring 1 Sun Apr 2:00 last Sun Oct 2:00
                                  033: ip subnet-zero
                                  034: no ip source-route
                                  035: ip tcp synwait-time 10
                                  036: !
                                  037: !
                                  038: ip cef
                                  039: !
                                  040: !
                                  041: ip flow-cache timeout active 1
                                  042: no ip bootp server
                                  043: ip domain name yourdomain.com
                                  044: ip name-server <ip>
                                  045: ip name-server <ip>
                                  046: ip name-server <ip>
                                  047: ip name-server <ip>
                                  048: ip name-server <ip>
                                  049: !
                                  050: username Administrator privilege 15 secret 5 <secrets>
                                  051: !
                                  052: !
                                  053: !
                                  054: interface Null0
                                  055: no ip unreachables
                                  056: !
                                  057: interface FastEthernet0/0
                                  058: description $ETH-SW-LAUNCH$$INTF-INFO-FE 0/0$$ETH-LAN$$FW_INSIDE$
                                  059: ip address 172.16.100.247 255.255.0.0
                                  060: ip access-group 101 in
                                  061: no ip redirects
                                  062: no ip unreachables
                                  063: no ip proxy-arp
                                  064: ip flow ingress
                                  065: ip flow egress
                                  066: ip nat inside
                                  067: ip route-cache flow
                                  068: duplex auto
                                  069: speed auto
                                  070: no mop enabled
                                  071: !
                                  072: interface FastEthernet0/1
                                  073: description $FW_OUTSIDE$
                                  074: ip address dhcp
                                  075: ip access-group sdm_fastethernet0/1_in in
                                  076: no ip redirects
                                  077: no ip unreachables
                                  078: no ip proxy-arp
                                  079: ip flow ingress
                                  080: ip flow egress
                                  081: ip nat outside
                                  082: ip route-cache flow
                                  083: duplex auto
                                  084: speed auto
                                  085: no mop enabled
                                  086: !
                                  087: ip classless
                                  088: ip route 172.19.0.0 255.255.0.0 172.16.200.40
                                  089: ip flow-export source FastEthernet0/0
                                  090: ip flow-export version 5
                                  091: ip flow-export destination 172.16.121.22 2055
                                  092: ip flow-top-talkers
                                  093: top 10
                                  094: sort-by bytes
                                  095: cache-timeout 500
                                  096: !
                                  097: ip http server
                                  098: ip http access-class 1
                                  099: ip http authentication local
                                  100: ip http timeout-policy idle 60 life 86400 requests 10000
                                  101: ip nat inside source list 2 interface FastEthernet0/1 overload
                                  102: ip nat inside source static tcp 172.16.121.77 80 12.201.24.128 80 extendable
                                  103: ip nat inside source static tcp 172.16.121.77 443 12.201.24.128 443 extendable
                                  104: ip nat inside source static tcp 172.16.121.77 1494 12.201.24.128 1494 extendable
                                  105: ip nat inside source static udp 172.16.121.77 1604 12.201.24.128 1604 extendable
                                  106: !
                                  107: ip access-list extended sdm_fastethernet0/1_in
                                  108: remark SDM_ACL Category=1
                                  109: remark SolarWinds
                                  110: permit udp any host 172.16.121.22 eq 2055
                                  111: remark SolarWinds 2
                                  112: permit udp any host 172.16.121.22 eq 9995
                                  113: remark citrix ica client udp
                                  114: permit udp any host 172.16.121.77 eq 1604 log
                                  115: remark citrix ica client
                                  116: permit tcp any host 172.16.121.77 eq 1494 log
                                  117: remark citrix ssh
                                  118: permit tcp any host 172.16.121.77 eq 443 log
                                  119: remark citrix www
                                  120: permit tcp any host 172.16.121.77 eq www log
                                  121: remark permit all
                                  122: permit ip any any
                                  123: !
                                  124: logging 172.16.159.2
                                  125: logging 172.16.121.22
                                  126: access-list 1 permit 172.16.100.245
                                  127: access-list 1 permit 172.16.100.133
                                  128: access-list 1 remark HTTP Access-class list
                                  129: access-list 1 remark SDM_ACL Category=1
                                  130: access-list 1 permit 172.16.100.87
                                  131: access-list 1 permit 172.16.100.1
                                  132: access-list 1 permit 172.16.100.2
                                  133: access-list 1 permit 172.16.100.3
                                  134: access-list 1 permit 172.16.0.0 0.0.255.255
                                  135: access-list 1 deny any
                                  136: access-list 2 remark SDM_ACL Category=2
                                  137: access-list 2 permit 172.16.0.0 0.0.255.255
                                  138: access-list 2 permit 172.19.0.0 0.0.255.255
                                  139: access-list 100 remark VTY Access-class list
                                  140: access-list 100 remark SDM_ACL Category=1
                                  141: access-list 100 permit ip host 172.16.100.87 any
                                  142: access-list 100 permit ip host 172.16.100.1 any
                                  143: access-list 100 permit ip host 172.16.100.2 any
                                  144: access-list 100 permit ip host 172.16.100.133 any
                                  145: access-list 100 permit ip host 172.16.100.3 any
                                  146: access-list 100 permit ip host 172.16.100.245 any
                                  147: access-list 100 permit ip 172.16.0.0 0.0.255.255 any
                                  148: access-list 100 deny ip any any
                                  149: access-list 101 remark Auto generated by SDM Management Access feature
                                  150: access-list 101 remark SDM_ACL Category=1
                                  151: access-list 101 remark SolarWinds
                                  152: access-list 101 permit udp any host 172.16.100.247 eq 2055
                                  153: access-list 101 permit tcp host 172.16.100.87 host 172.16.100.246 eq telnet
                                  154: access-list 101 permit tcp host 172.16.100.1 host 172.16.100.246 eq telnet
                                  155: access-list 101 permit tcp host 172.16.100.2 host 172.16.100.246 eq telnet
                                  156: access-list 101 permit tcp host 172.16.100.133 host 172.16.100.246 eq telnet
                                  157: access-list 101 permit tcp host 172.16.100.3 host 172.16.100.246 eq telnet
                                  158: access-list 101 permit tcp host 172.16.100.245 host 172.16.100.246 eq telnet
                                  159: access-list 101 permit tcp host 172.16.100.87 host 172.16.100.246 eq 22
                                  160: access-list 101 permit tcp host 172.16.100.1 host 172.16.100.246 eq 22
                                  161: access-list 101 permit tcp host 172.16.100.2 host 172.16.100.246 eq 22
                                  162: access-list 101 permit tcp host 172.16.100.133 host 172.16.100.246 eq 22
                                  163: access-list 101 permit tcp host 172.16.100.3 host 172.16.100.246 eq 22
                                  164: access-list 101 permit tcp host 172.16.100.245 host 172.16.100.246 eq 22
                                  165: access-list 101 permit tcp host 172.16.100.87 host 172.16.100.246 eq www
                                  166: access-list 101 permit tcp host 172.16.100.1 host 172.16.100.246 eq www
                                  167: access-list 101 permit tcp host 172.16.100.2 host 172.16.100.246 eq www
                                  168: access-list 101 permit tcp host 172.16.100.133 host 172.16.100.246 eq www
                                  169: access-list 101 permit tcp host 172.16.100.3 host 172.16.100.246 eq www
                                  170: access-list 101 permit tcp host 172.16.100.245 host 172.16.100.246 eq www
                                  171: access-list 101 permit tcp host 172.16.100.87 host 172.16.100.246 eq cmd
                                  172: access-list 101 permit tcp host 172.16.100.1 host 172.16.100.246 eq cmd
                                  173: access-list 101 permit tcp host 172.16.100.2 host 172.16.100.246 eq cmd
                                  174: access-list 101 permit tcp host 172.16.100.133 host 172.16.100.246 eq cmd
                                  175: access-list 101 permit tcp host 172.16.100.3 host 172.16.100.246 eq cmd
                                  176: access-list 101 permit tcp host 172.16.100.245 host 172.16.100.246 eq cmd
                                  177: access-list 101 permit udp host 172.16.100.87 host 172.16.100.246 eq snmp
                                  178: access-list 101 permit udp host 172.16.100.1 host 172.16.100.246 eq snmp
                                  179: access-list 101 permit udp host 172.16.100.2 host 172.16.100.246 eq snmp
                                  180: access-list 101 permit udp host 172.16.100.133 host 172.16.100.246 eq snmp
                                  181: access-list 101 permit udp host 172.16.100.3 host 172.16.100.246 eq snmp
                                  182: access-list 101 permit udp host 172.16.100.245 host 172.16.100.246 eq snmp
                                  183: access-list 101 deny tcp any host 172.16.100.246 eq telnet
                                  184: access-list 101 deny tcp any host 172.16.100.246 eq 22
                                  185: access-list 101 deny tcp any host 172.16.100.246 eq www
                                  186: access-list 101 deny tcp any host 172.16.100.246 eq 443
                                  187: access-list 101 deny tcp any host 172.16.100.246 eq cmd
                                  188: access-list 101 deny udp any host 172.16.100.246 eq snmp
                                  189: access-list 101 permit ip any any
                                  190: snmp-server community <community string> RW
                                  191: no cdp run
                                  192: !
                                  193: control-plane
                                  194: !
                                  195: banner login ^C
                                  196: -----------------------------------------------------------------------
                                  197: Cisco Router and Security Device Manager (SDM) is installed on this device.
                                  198: This feature requires the one-time use of the username "cisco"
                                  199: with the password "cisco". The default username and password have a privilege level of 15.
                                  200: Please change these publicly known initial credentials using SDM or the IOS CLI.
                                  201: Here are the Cisco IOS commands.
                                  202: username privilege 15 secret 0
                                  203: no username cisco
                                  204: Replace and with the username and password you want to use.
                                  205: For more information about SDM please follow the instructions in the QUICK START
                                  206: GUIDE for your router or go to http://www.cisco.com/go/sdm
                                  207: -----------------------------------------------------------------------
                                  208: ^C
                                  209: !
                                  210: line con 0
                                  211: login authentication local_authen
                                  212: transport output telnet
                                  213: line aux 0
                                  214: login authentication local_authen
                                  215: transport output telnet
                                  216: line vty 0 4
                                  217: access-class 100 in
                                  218: authorization exec local_author
                                  219: login authentication local_authen
                                  220: transport input telnet
                                  221: line vty 5 15
                                  222: access-class 100 in
                                  223: authorization exec local_author
                                  224: login authentication local_authen
                                  225: transport input telnet
                                  226: !
                                  227: scheduler allocate 20000 1000
                                  228: !
                                  229: end
                                  230: 
                                  231: 

                        • Re: Giving up on NTA - Last Ditch Efforts
                          viperar15

                          have you verified that the NetFlow Collector Service is "UP" on the NetFlow Page for NPM?

                          Should be something like this...

                          ServerName  Status  Port
                          Server1         Up         2055

                          If it is not.... you have a problem on NTA Collector itself.

                          The other thing might be a Windows Firewall Problem or IPSEC if you're using that.

                            • Re: Giving up on NTA - Last Ditch Efforts
                              Andy McBride

                              Jeremy,

                              can you post the output of sho ip flo exp and sho ip cache verbose flo

                              Andy

                                • Re: Giving up on NTA - Last Ditch Efforts

                                  I have not yet contacted Cisco TAC and the NetFlow Collector Service is UP.  I can ping 172.16.121.22 (the collector) from the router.  The Windows Firewall on the collector has all of the necessary exceptions (as verified by support; also verified by successfully sending generic NetFlow data to the collector from an unmanaged device)  Thanks to everyone for all of the feedback.  Also, please let me know if more information is required from the verbose command.

                                  Show IP Flow Export yields:

                                  Flow export v5 is enabled for main chache
                                    Exporting flows to 172.16.121.22 (2055)
                                    Exporting using source interface FastEthernet0/0
                                    Version 5 flow records
                                    64401130 flows exported in 2151625 udp datagrams
                                    0 flows failed due to lack of export packet
                                    0 export packets were sent up to process level
                                    0 export packets were dropped due to no fib
                                    0 export packets were dropped due to adjacency issues
                                    0 export packets were dropped due to fragmentation failures
                                    0 export packets were dropped due to encapsulation fixup failures

                                  sho ip cache verbose flo yields:

                                  IP packet size distribution (1543M total packets):
                                     1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
                                     .000 .431 .023 .005 .004 .003 .003 .005 .004 .003 .004 .003 .002 .003 .002

                                      512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
                                     .003 .002 .053 .022 .415 .000 .000 .000 .000 .000 .000

                                  IP Flow Switching Cache, 278544 bytes
                                    275 active, 3821 inactive, 81426648 added
                                    933428740 ager polls, 0 flow alloc failures
                                    Active flows timeout in 1 minutes
                                    Inactive flows timeout in 15 seconds
                                  IP Sub Flow Cache, 21640 bytes
                                    275 active, 749 inactive, 64409336 added, 64409336 added to flow
                                    0 alloc failures, 12615 force free
                                    1 chunk, 4169 chunks added
                                    last clearing of statistics never
                                  Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
                                  --------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
                                  TCP-Telnet        3703      0.0        31   247      0.0      13.5      12.9
                                  TCP-FTP         115336      0.0         3    54      0.0       0.6      15.0
                                  TCP-FTPD           136      0.0       391   893      0.0       2.2       2.9
                                  TCP-WWW       63715701     14.8        19   717    288.6       2.2       6.4
                                  TCP-SMTP          7554      0.0         7    85      0.0       1.8       3.3
                                  TCP-X              179      0.0         6   716      0.0       0.6      14.4
                                  TCP-NNTP             1      0.0         1    40      0.0       0.0       1.1
                                  TCP-Frag           279      0.0        30    27      0.0      11.2      15.5
                                  TCP-other      7022222      1.6        37   690     60.7       3.6       6.5
                                  UDP-DNS         257883      0.0         1    66      0.0       1.7      15.4
                                  UDP-NTP         351078      0.0         1    76      0.0       0.0      15.4
                                  UDP-TFTP         12011      0.0         5    49      0.0      20.0      15.4
                                  UDP-Frag           931      0.0         2   660      0.0       0.1      15.6
                                  UDP-other      9742115      2.2         4   238      9.5       1.9      15.4
                                  ICMP            196462      0.0         1    68      0.0       0.4      15.5
                                  Total:        81425591     18.9        18   699    359.3       2.3       7.6

                                  SrcIf          SrcIPaddress    DstIf          DstIPaddress    Pr TOS Flgs  Pkts
                                  Port Msk AS                    Port Msk AS    NextHop              B/Pk  Active