20 Replies Latest reply: Dec 14, 2012 6:06 AM by Choly RSS

Nodes Traffic problem

rulob

Good morning,

 

I've recently taken over the management of the Solarwinds platform installed in this company. They've been using it for some time and they let it running after some basic configs and installs.

 

Now is my job to patch it properly and let it running in prime condition. However ive encounter some problems, for one with NTA(problems on our part im sure).

As you can see in the pic, most of the nodes addes have stopped sending data or have never sent it.

 

The config of those routers are all the same, and idk why is that they stopped sending data. If there is any pointer as to where to look up I'll appreciate it.

 

Happy monitoring,

 

Raul.

 
  • Re: Nodes Traffic problem
    mharvey

    Raul,

     

    Have you confirmed within the configs that that destination is the IP of the Orion NTA server and being sent to port 2055?  Did the Orion server get migrated at all?  If so, the IP for the destination may not be correct and flows may be going to an older server. 

     

    Regards,

    Matthew Harvey

    Loop1 Systems

    http://www.loop1systems.com

  • Re: Nodes Traffic problem
    rulob

    So, I tried doing some sniffing with Wireshark, but Im not sure whats going on.

     

    As you see in the 1st screen shot, the last node sent some Netflow data at 11:03am. However, WireShark never got a cflow packet, only lots of SNMP packets. You'll see this in the second screenshot and also please notice I only put the filter for the IP of the node.

     

    If you guys have any ideas, i'll appreciate the input.

    nodes.png

    wshark.PNG

     

    P.S. There is something wrong with the web site and Chrome. I had to use IE to be able to use the Insert Image button. With Chrome I only had a black window open, but no text or buttons of any prompt.

    • Re: Nodes Traffic problem
      Choly

      Well, as Wireshark doesn't see any cflow packets coming you have to find a cause somewhere on your network. SNMP packets are related to Orion monitoring that device.

       

      Could you post here NetFlow related part of your device config?

  • Re: Nodes Traffic problem
    rulob

    Hello,

     

    I put this on standby for a bit and didnt do much till yesterday.

     

    I set the timeouts as Choly said, but this didnt helped either. Now my thoughts are that maybe is the firewall which is giving me some problems. I set a new rule in the firewall that should let pass all UDP packets to port 2055 from the router I want to monitor.

     

    Is there something I am missing in the firewall?

  • Re: Nodes Traffic problem
    rulob

    Hello again,

     

    Well, I think I found a main problem. and is regarding my device. I run some commands to show the flow export from my device and found a disturbing issue.

     

    Router#sho ip flow exp

    Flow export v5 is enabled for main cache

      Exporting flows to 10.90.XXX.XXX (2055)

      Exporting using source interface Multilink25

      Version 5 flow records

      29253320 flows exported in 975381 udp datagrams

      0 flows failed due to lack of export packet

      0 export packets were sent up to process level

      0 export packets were dropped due to no fib

      25 export packets were dropped due to adjacency issues

      0 export packets were dropped due to fragmentation failures

      0 export packets were dropped due to encapsulation fixup failures

     

    Router#sho ip flow int

    Multilink25

      ip route-cache flow

      ip flow ingress

      ip flow egress

     

    What I find a bit disturbing is the amount of flows and UDP Datagrams, is this normal? Secondly, those 25 packets dropped, what it means? I have to assume that no matter that only 25 dropped since I have so many flows and datagrams sent right?

    Lastly, I think the interface is correctly configured. If anything is wrong I'd appreciate an input

     

    Regards,

     

    --Raul

    • Re: Nodes Traffic problem
      jswan

      The flow export and datagram counts look normal to me.

       

      The "25 export packets were dropped due to adjacency issues" means that 25 flow export packets were lost due to some transient routing problem on the device that caused it to lose its route to the Orion server (probably an interface flap).

       

      Since your Orion server isn't receiving the flow packets, that's the thing you need to troubleshoot:

       

      1) Can you ping the Orion server from the IP address of Multilink25?

      2) Is there a firewall or other packet filter in the middle that's blocking UDP/2055?

      • Re: Nodes Traffic problem
        rulob

        Hi jswan,

         

        As I posted above the post you replied to, I have a Firewall. Also, I have set a rule to allow packets from this router to our Orion server on port 2055.

         

        I havent tried the ping, but last night cheking Wireshark, I saw the ICMP packets beetween the server and the router, getting Request and Response, so I assume its reacheable.

         

        In a very old post I saw that it should be the loopback of the router the interface i should be monitoring? Has anyone else tried this?

         

        Regards, Raul

        • Re: Nodes Traffic problem
          jswan

          Not sure about the current release, but in earlier releases the node IP address in NPM had to be the same as the NetFlow source address. But since you're not receiving netflow packets on the NPM host at all, that's not your problem. Until you are seeing NetFlow packets received on the NPM host, you're not going to get anywhere.

        • Re: Nodes Traffic problem
          mharvey

          Something else to look at.  On the Orion server, go to Start > Run > Perfmon.  Open the Performance Monitor and click the + then add all the SolarWinds NetFlow counters.  Once added, change the view from a graph to a report, then see if the server is seeing the flow packets.  While you aren't seeing them as FLOW or CFLOW in Wireshark, they may be coming in simply as UDP packets and Wireshark isn't recognizing them.

          • Re: Nodes Traffic problem
            rulob

            Thanks for all the input guys.

             

            Some good news, of the 3 nodes that weren't sending, now I recieve from one more, that puts me on 2/4 for Netflow. Now, probably what helped me, is that this 2 nodes are on-premises, so they don't have to pass by the Firewall. However the other 2 nodes are...elsewhere in the country so they do need to pass the Firewall.

             

            Currently checking the configs of the other nodes and see what I can get.

             

            Will keep posted.

  • Re: Nodes Traffic problem
    rulob

    Hi jswan,

     

    In an earlier thread you posted the following:

     

              "Most edge routers are configured to do dynamic NAT (aka PAT) on the outside interface. If this is the case with yours, you'll need to      configure a static NAT translation on the edge router for the outside devices to talk to the NTA server. It's not enough to just allow the NDE packets through the ACL unless you're not doing NAT."

     

    Now, as I understand, unless the Firewall is doing NAT, you have to put the outisde interface of the Firewall as the destination, correct?

     

    Since this is not my case a basic rule of allowing packets to pass should be enough so the Firewall let the packets coming from my exporter pass.

     

    My topology is kind of like this.:

    Outside Exporter -> Firewall -> Router(NAT) -> Internal Netwrok ->Orion Server(NPM,NTA,etc)

     

    As you can see, is the router not the firewall doing the NAT. Maybe I am missing something here, but clearly I have problems seeing it.

    • Re: Nodes Traffic problem
      jswan

      If the outside exporter is sending the netflow packets with an "outside" source address (e.g., a public Internet address), then you would need to configure a translation on your NAT router that translates the destination address to the Orion server's address. Your firewall will also need to be configured to allow the packets.

       

      For example (firewall is NOT doing NAT):

       

      R1 (1.1.1.1 outside interface) ---> Firewall (permit udp/2055 from 1.1.1.1 to 2.2.2.2) ---> R2 (xlate 2.2.2.2 to 10.1.1.1) ---> Orion (10.1.1.1)

       

      In words:

      1. R1 generates a netflow export packet to UDP port 2055 with src IP 1.1.1.1 and dst IP 2.2.2.2.
      2. Firewall permits UDP/2055 from 1.1.1.1 to 2.2.2.2 and routes it to the outside interface of R2.
      3. R2 has a correctly configured NAT rule to translate the destination address from 2.2.2.2 to 10.1.1.1.
      4. Orion receives the packet from 1.1.1.1 on UDP/2055.

       

      This is how you would do it if you need to use public IP addresses to source your NetFlow packets.