3 Replies Latest reply on Aug 6, 2012 2:02 PM by jwhitten

    How to Use NPROBE / Netflow With Linux -- And a Question

    jwhitten

      Howdy,

       

      First I want to thank Solarwinds for all their great work, the moderators and contributors here on Thwack for all their great questions and helpful answers, and the folks at Ntop for their really nice Netflow collector product "NProbe". Nprobe works really well and plays very nicely with Solarwinds NTA. But, as many folks (including myself) have discovered, it can be a royal pain to figure out and set up. I am not an expert at Netflow (nor NProbe) by any means, but I wanted a better way to configure and manage NProbe collections in our environment. For us it is an essential item of operation and as such merits a decent mechanism to control it.

       

      First a little bit about our environment... In our setup we have a number of network "Zones" that we need to monitor and track usage stats for. We'll concentrate on just four of them in this discussion and we'll call them "DMZ", "USER", "SERVER" and "COREX", and assign them the (scrubbed) net-blocks of 2.2.2.0/24, 3.3.3.0/24, 4.4.4.0/24 and 5.5.5.0/24 respectively, and one additional network 1.2.3.4 for the collection server itself (which we call "MARS-NPROBE").

       

      After purchasing the NProbe software from Ntop.org, compiling it and getting it set up on our system, I went through the usual routine of trying out all the options (including many iterations of the '-u' and '-Q' parameters) and finally got to the point where it seemed like it was pretty much working but I was still occasionally tweaking it a bit here and there, which involved a lot of starting, stopping, "ps"-ing and "grep"-ing over and over ad-naseum until I thought "there's got to be a better way"... what follows are the collection of setup and management scripts that I wrote for our organization. Perhaps they'll help you get started with NProbe and Netflow monitoring in yours...

       

      It turns out that all those "configuration" options in "/etc/sysconfig/network-scripts" (e.g. "ifcfg-eth0", et al) are really nothing more than a collection of shell script variables that are set in a convenient, well-known place, that the actual interface control scripts ("ifup" & "ifdown") can use to properly configure the respective NIC port on the system. I kept thinking about those config files and thinking what a good place they make to put the Nprobe configuration if I could just figure out how to access them at the right time. My first attempt was a script that simply knew about the "/etc/sysconfig/network-scripts" location and read the various ifcfg files directly. Then I was poking around in the "ifup" control script and realized that it actually had a provision for local actions-- namely the "/sbin/ifup-local" and "/sbin/ifdown-local" scripts-- which I quickly realized would be the ideal mechanism for launching the Nprobe-specific control scripts. That way Nprobe would be started and/or stopped via the normal and customary action of starting or stopping the NIC port itself with no further intervention required. And being a rather lazy admin myself, that seemed like the perfect opportunity for an "elegant solution" AND a little less work on my part! A double win if there ever was one, right? :-)

       

      Originally I was numbering my collection ports-- putting a numbered interface into each collection environment. Then it dawned on me that if the interface is promiscuous, it doesn't really need a number since it's going to see everything on the interface anyway. So you'll notice in the configuration scripts that the usual network configuration information for the collection ports is all commented-out. Also you will notice several new NProbe-specific options which are useful for passing information to the NProbe program-- such as a nice "NAME" for the NIC-- its helpful when you're looking through a "ps -ef | grep nprobe" output to be able to see which nprobe process is monitoring which zone. You can also pass information to describe the network to be monitored, etc. There are additional NProbe setup and configuration options which can be set in the NProbe control script itself, which lives in "/usr/local/sbin/nprobe-ctl.sh". I realize it's strictly not a stripped-binary, but sbin still seemed like an appropriate place to put it. You can put it wherever you like, just be sure to update the other scripts accordingly.

       

      Over time, I have used several methods of communicating with the Solarwinds Orion NPM / NTA server. One very useful method, which I highly recommend if you have a spare NIC, is to use a dedicated NIC with a direct "crossover" style cable and just go port-to-port. If you do this, you only need a "private interface" set up between you and the other system-- so you can set the NIC ports to whatever private addresses you want, just be sure to make them on the same network. Presently, I am using the regular interface on the system and that works pretty well too. But it does put traffic back out on the switching network and so is technically disturbing the observation and contributing a little bit to the overhead on the network. I'll get around to reinstalling the crossover cable at some point, but I just wanted to point out that you can do it either way with acceptable results.

       

      The other options in the file should, hopefully, be self-explanatory. And if not, it should only take a little sleuthing through the scripts to figure them out. But I do want to talk briefly about setting up the monitor sessions on the switches. We use Cisco gear so the monitor examples are specific to those, but the idea is similar and available on other types of switches. I've also included a sample "monitor session" configuration for a typical Cisco switch so you can see how that's done. The biggest thing to watch out for is that you don't include your monitor ("destination") port as one of the source ports. It may be that IOS won't actually let you do that so it may not be a big concern. For other switches you'll need to read the manual.

       

      Now, here's my question-- it doesn't have anything to do with the "control" aspect of the Nprobe stuff, but rather back to those pesky '-u' and '-Q' NProbe options... I recently discovered that I only seem to be collecting INBOUND traffic-- at least from the perspective of Solarwinds NTA. There are only ingress stats available. Does this seem right? Or should I be seeing two streams-- an "Inbound" and an "Outbound" stream? If so, how do I achieve that?

       

      Okay-- here are the scripts. Copy them to the locations indicated, adjust the settings to match your environment, and then they should just work. Don't forget to set the control scripts executable.

       

       

      [mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth2

       

      ##--------------------------------------

      ## MARS-NPROBE-USER

      ##--------------------------------------

      ##

      DEVICE=eth2

      NAME="MARS-NPROBE-USER"

      HWADDR=00:11:22:33:8B:D0

       

      NM_CONTROLLED=no

      ONBOOT=yes

      TYPE=Ethernet

      BOOTPROTO=none

      IPV4_FAILURE_FATAL=no

      IPV6INIT=no

      USERCTL=no

      DEFROUTE=no

       

      ## Basic Network Settings

      #IPADDR=2.2.2.42

      #NETWORK=2.2.2.0

      #NETMASK=255.255.255.0

      #GATEWAY=2.2.2.254

       

      ## Automagically fire-up NProbe

      PROMISC=yes

      NPROBE=yes

      ## Must be set for nprobe!

      CIDR="2.2.2.0/24"

      ## Set this to ethNN

      INDEX="2"

       

       

      [mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth3

       

      ##--------------------------------------

      ## MARS-NPROBE-SERVER

      ##--------------------------------------

      ##

      DEVICE=eth3

      NAME="MARS-NPROBE-SERVER"

      HWADDR=00:11:22:33:8B:D1

       

      NM_CONTROLLED=no

      ONBOOT=yes

      TYPE=Ethernet

      BOOTPROTO=none

      IPV4_FAILURE_FATAL=no

      IPV6INIT=no

      USERCTL=no

      DEFROUTE=no

       

      ## Basic Network Settings

      #IPADDR=3.3.3.42

      #NETWORK=3.3.3.0

      #NETMASK=255.255.255.0

      #GATEWAY=3.3.3.254

       

      ## Automagically fire-up NProbe

      PROMISC=yes

      NPROBE=yes

      ## Must be set for nprobe!

      CIDR="3.3.3.0/24"

      ## Set this to ethNN

      INDEX="3"

       

       

      [mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth4

       

      ##--------------------------------------

      ## MARS-NPROBE-DMZ

      ##--------------------------------------

      ##

      DEVICE=eth4

      NAME="MARS-NPROBE-DMZ"

      HWADDR=00:11:22:33:8B:D4

       

      NM_CONTROLLED=no

      ONBOOT=yes

      TYPE=Ethernet

      BOOTPROTO=none

      IPV4_FAILURE_FATAL=no

      IPV6INIT=no

      USERCTL=no

      DEFROUTE=no

       

      ## Basic Network Settings

      #IPADDR=4.4.4.42

      #NETWORK=4.4.4.0

      #NETMASK=255.255.255.0

      #GATEWAY=4.4.4.1

       

      ## Automagically fire-up NProbe

      PROMISC=yes

      NPROBE=yes

      ## Must be set for nprobe!

      CIDR="4.4.4.0/24"

      ## Set this to ethNN

      INDEX="4"

       

       

      [mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth5

       

      ##--------------------------------------

      ## MARS-NPROBE-COREX

      ##--------------------------------------

      ##

      DEVICE=eth5

      NAME="MARS-NPROBE-COREX"

      HWADDR=00:11:22:33:8B:D5

       

      NM_CONTROLLED=no

      ONBOOT=yes

      TYPE=Ethernet

      BOOTPROTO=none

      IPV4_FAILURE_FATAL=no

      IPV6INIT=no

      USERCTL=no

      DEFROUTE=no

       

      ## Basic Network Settings

      #IPADDR=5.5.5.42

      #NETWORK=5.5.5.0

      #NETMASK=255.255.255.0

      #GATEWAY=5.5.5.1

       

      ## Automagically fire-up NProbe

      PROMISC=yes

      NPROBE=yes

      ## Must be set for nprobe!

      CIDR="5.5.5.0/24"

      ## Set this to ethNN

      INDEX="5"

       

       

      [mars-nprobe sbin]# cat /sbin/ifup-local

       

      #!/bin/sh

       

      ## ifup-local -- JWHITTEN -- 10/03/2011

      ## Script to handle local modifications to 'ifup' process

       

      cd /etc/sysconfig/network-scripts

      . ./network-functions

       

      [ -f ../network ] && . ../network

       

      unset REALDEVICE

      if [ "$1" = --realdevice ] ; then

          REALDEVICE=$2

          shift 2

      fi

       

      CONFIG=$1

       

      need_config "${CONFIG}"

       

      source_config

       

      ## see if we want PROMISC (Promiscuous) mode and turn it on

       

      #echo "IFUP-LOCAL: CONFIG=${CONFIG}"

      #echo "PROMISC WANTED? ${PROMISC}"

       

      if [ "$PROMISC" = yes ]; then

      #      echo "COMMAND=/sbin/ifconfig ${DEVICE} promisc"

              /sbin/ifconfig ${DEVICE} promisc

      fi

       

       

      ## see if we need to fire up an nprobe for this port

       

      if [ "${NPROBE}" = yes ]; then

              /usr/local/sbin/nprobe-ctl.sh start "${DEVICE}" "${NAME}"

      fi

       

       

      [mars-nprobe sbin]# cat /sbin/ifdown-local

       

      #!/bin/sh

       

      ## ifdown-local -- JWHITTEN -- 10/03/2011

      ## Script to handle local modifications to 'ifdown' process

       

      cd /etc/sysconfig/network-scripts

      . ./network-functions

       

      [ -f ../network ] && . ../network

       

      unset REALDEVICE

      if [ "$1" = --realdevice ] ; then

          REALDEVICE=$2

          shift 2

      fi

       

      CONFIG=$1

       

      need_config "${CONFIG}"

       

      source_config

       

      ## see if we want PROMISC (Promiscuous) mode and turn it off

       

      #echo "IFDOWN-LOCAL: CONFIG=${CONFIG}"

      #echo "PROMISC WANTED? ${PROMISC}"

       

      if [ "$PROMISC" = yes ]; then

      #      echo "COMMAND=/sbin/ifconfig ${DEVICE} -promisc"

              /sbin/ifconfig ${DEVICE} -promisc

      fi

       

       

      ## see if we need to shut down an nprobe for this port

       

      if [ "${NPROBE}" = yes ]; then

              /usr/local/sbin/nprobe-ctl.sh stop "${DEVICE}" "${NAME}"

      fi

       

       

      [mars-nprobe sbin]# cat /usr/local/sbin/nprobe-ctl.sh

       

      #! /bin/sh

       

      #############################################################################

      ## RUN_NPROBES.SH -- revision 0.1 -- JWHITTEN -- 25-AUG-2011

      ## This is a first stab at a startup script for the Nprobe / NetFlow

      ## processing engine Nprobe processes SPAN/RSPAN traffic from switches

      ## and converts it into NetFlow traffic for the monitoring and security

      ## environments.

      ##

      ## At some point it will be necessary to revisit this script and add

      ## additional Nprobe instanaces to monitor additional incoming traffic

      ## (i.e., additional ethernet ports). Also the plan is to re-broadcast

      ## the Nprobe / NetFlow feed to OSSIM and Snort on direct, dedicated

      ## ethernet ports.

      #############################################################################

      ##

       

      ##############################################################################

      ## Define some variables

      ##############################################################################

      ##

       

      ## Where PID files live (usually /var/run)

      PID_DIR="/var/run"

       

      ## Where is the Nprobe binary located?

      CMD_NPROBE="/usr/local/bin/nprobe"

       

      ## Which interface shall we monitor?

      IFACE="eth0"

       

      ## MARS-ORION Dedicated Port

       

      ## Use with dedicated crossover cable

      ## COLLECTION_HOST="10.254.254.254"

       

      ## use if sent over the network

      COLLECTION_HOST="1.2.3.4"

       

      ## Normal Port for NetFlow / NProbe

      COLLECTION_PORT="2055"

       

      ## Beginning Offset into SNMP ID Table? (usually 1)

      SNMP_IDX_START=1

       

      ## Ending Offset into SNMP ID Table? (perhaps 2? Not sure what this should be..??)

      SNMP_IDX_END=2

       

      ## Prefix for Probe Instance Name

      DEFAULT_INSTANCE_PREFIX="MARS-NPROBE"

       

      ## NProbe Single Opts

      NPROBE_OPTS="-G -a"

       

      ## Where nprobe stats files live

      STATS_DIR="/var/www/html/nprobe"

       

      ##============================================================================

      ## ****  N O T I C E  --  N O T I C E  --  N O T I C E  --  N O T I C E  ****

      ## ****      NO USER SERVICEABLE PARTS BELOW THIS LINE !!!              ****

      ##============================================================================

       

       

      ##############################################################################

      ## Functions & Subroutines

      ##############################################################################

      ##

       

      ## load device config file

      function need_device_config() {

              CONFIG=${1}

       

              cd /etc/sysconfig/network-scripts

              . ./network-functions

       

              [ -f ../network ] && . ../network

       

              unset REALDEVICE

              if [ "$1" = --realdevice ] ; then

                  REALDEVICE=$2

                  shift 2

              fi

       

       

              need_config "${CONFIG}"

       

              source_config

              }

       

      ## construct an nprobe instance name

      function need_instance_name() {

              IFACE=${1}

       

              ## convert to upper case for display version

              D_IFACE=`echo ${IFACE} | awk '{ print toupper($0) }'`

       

              ## How this instance will be referenced in the syslog

              if [ "x${NAME}" = "x" ]; then

                      INSTANCE_NAME="${D_IFACE}-${DEFAULT_INSTANCE_PREFIX}"

              else

                      INSTANCE_NAME="${D_IFACE}-${NAME}"

              fi

       

              return

              }

       

      function need_bound_network() {

              BOUND_NETWORK="${CIDR}@${INDEX}"

              }

       

      function need_pid_file_name() {

              INSTANCE_NAME=${1}

              PID_FILE="${PID_DIR}/${INSTANCE_NAME}.pid"

              }

       

      function need_stats_file_name() {

              INSTANCE_NAME=${1}

              STATS_FILE="${STATS_DIR}/${INSTANCE_NAME}.txt"

              }

       

      ## log an action to the syslog

      function log_action() {

              ACTION=${1}

              /usr/bin/logger "${ACTION}"

              }

       

      ## start up an nprobe

      function nprobe_start() {

              ## Our command line

              IFACE=${1}

       

              ## see if we want a stats file

              if [ "x${STATS_DIR}" != "x" ]; then

                      need_stats_file_name "${INSTANCE_NAME}"

                      OPT_STATS_FILE="-9 ${STATS_FILE}"

              fi

       

              ## see if we want to identify the network

       

              NPROBE_PRIMARY_OPTS="${NPROBE_OPTS} -i ${IFACE} -n ${COLLECTION_HOST}:${COLLECTION_PORT} -I ${INSTANCE_NAME} -g ${PID_FILE}"

       

              ## bundle-up the snmp index opts

      #      OPT_SNMP_IDX="-u ${SNMP_IDX_START} -Q ${SNMP_IDX_END}"

              OPT_SNMP_IDX="-u ${INDEX} -Q ${INDEX}"

       

              ## set up network binding for port/network identification

              need_bound_network

              OPT_BOUND_NETWORK="-L ${BOUND_NETWORK}"

       

              ## assemble nprobe command

              CMD="${CMD_NPROBE} ${NPROBE_PRIMARY_OPTS} ${OPT_STATS_FILE} ${OPT_SNMP_IDX} ${OPT_BOUND_NETWORK}"

       

              echo "START COMMAND=${CMD}"

       

              ## execute the command

              ${CMD} &

       

              echo "Starting NProbe ${INSTANCE_NAME}"

       

              log_action "${CMD}"

              log_action "NProbe Instance ${INSTANCE_NAME} Started"

              }

       

      ## shutdown an nprobe

      function nprobe_stop() {

              IFACE=${1}

       

      #      echo $"Stopping ${INSTANCE_NAME}... (PID_FILE=${PID_FILE}) "

       

              MSG="Stopping NProbe ${INSTANCE_NAME} "

       

              ## if it exists, see if we can kill it

              if [ -f "${PID_FILE}" ]; then

                      kill 15 `cat ${PID_FILE}` && {

                              echo "${MSG} SUCCESS"

                              log_action "${MSG} SUCCESS"

                              } || {

                                      echo "${MSG} FAILED"

                                      log_action "${MSG} FAILED"

                                      }

      #      else

      #              echo $"Pid File for NProbe Instance ${INSTANCE_NAME} Not Found" >&2

              fi

              }

       

      function display_help() {

              ## doesn't return

              echo $"Usage: nprobe-ctl.sh [start|stop] <device name> {<nickname>} " >&2

              exit 1

              }

       

      ##############################################################################

      ## Begin Script

      ##############################################################################

      ##

       

      COMMAND=${1}; shift

      IFACE=${1}; shift

       

      [ -z "${COMMAND}" -o -z "${IFACE}" ] && {

              display_help

              }

       

      #echo "COMMAND=${COMMAND}, IFACE=${IFACE}"

       

      need_device_config "${IFACE}"

       

      need_instance_name "${IFACE}" "${NAME}"

      need_pid_file_name "${INSTANCE_NAME}"

       

      case "${COMMAND}" in

              start | START)

                      nprobe_start "${IFACE}"

                      ;;

              stop | STOP)

                      nprobe_stop "${IFACE}"

                      ;;

              *)

                      display_help

                      ;;

      esac

       

       

      This is what it looks like when its running...

      [mars-nprobe sbin]# ps -ef | grep nprobe

       

      nobody  29352    1  1 Jul23 ?        00:19:53 /usr/local/bin/nprobe -G -a -i eth2 -n 1.2.3.4:2055 -I ETH2-MARS-NPROBE-USER -g /var/run/ETH2-MARS-NPROBE-USER.pid -9 /var/www/html/nprobe/ETH2-MARS-NPROBE-USER.txt -u 2 -Q 2 -L 2.2.2.0/24@2

      nobody  29488    1  6 Jul23 ?        01:01:41 /usr/local/bin/nprobe -G -a -i eth3 -n 1.2.3.4:2055 -I ETH3-MARS-NPROBE-SERVER -g /var/run/ETH3-MARS-NPROBE-SERVER.pid -9 /var/www/html/nprobe/ETH3-MARS-NPROBE-SERVER.txt -u 3 -Q 3 -L 3.3.3.0/24@3

      nobody  29624    1  2 Jul23 ?        00:22:51 /usr/local/bin/nprobe -G -a -i eth4 -n 1.2.3.4:2055 -I ETH4-MARS-NPROBE-DMZ -g /var/run/ETH4-MARS-NPROBE-DMZ.pid -9 /var/www/html/nprobe/ETH4-MARS-NPROBE-DMZ.txt -u 4 -Q 4 -L 4.4.4.0/24@4

      nobody  29759    1  0 Jul23 ?        00:03:56 /usr/local/bin/nprobe -G -a -i eth5 -n 1.2.3.4:2055 -I ETH5-MARS-NPROBE-COREX -g /var/run/ETH5-MARS-NPROBE-COREX.pid -9 /var/www/html/nprobe/ETH5-MARS-NPROBE-COREX.txt -u 5 -Q 5 -L 5.5.5.0/24@5

       

       

      (On Cisco 3750 or 2960 Switch, the 'monitor session' portion from a 'show run'. Obviously you don't want to include the destination port in the source monitor.):

       

      monitor session 42 source interface Gi1/0/1 - 19 , Gi1/0/21 , Gi1/0/23 - 24

      monitor session 42 destination interface Gi1/0/22 encapsulation replicate

        • Re: How to Use NPROBE / Netflow With Linux -- And a Question
          mavturner

          Thanks for the great write up and information!

           

          I believe nProbe will also show as Ingress because that is how it sees the traffic based on its position in the network. I've asked the engineering team here to confirm.

          • Re: How to Use NPROBE / Netflow With Linux -- And a Question
            Martin.Krivanek

            nProbe takes two parameters (-u and -Q) to specify input and output interface index in flows it generates. For example -u 1 -Q 2 means that all flows will have 1 as input if index and 2 as output if index. From NTA perspective then all flows appear as ingress on first interface and egress on the other. Both interfaces on the machine where nProbe runs (index 1 and 2 in this example) have to be in Orion. If only one interface is polled by Orion, and I think this is the case, NTA won’t resolve if index 2 and all flows will only appear as ingress on first interface.

             

            But this doesn’t matter much, since information about original interfaces is lost with nProbe anyway. You would see exactly same traffic on both interfaces, it would just be ingress on one and egress on the other. This is only about viewing data, nothing is lost. So nProbe provides both direction, it’s just that flow direction is kind of useless with nProbe – all traffic appears as ingress on one interface and optionally as egress on the other.

             

            Let me know if you need more information.

             

            Regards,

            Martin

              • Re: How to Use NPROBE / Netflow With Linux -- And a Question
                jwhitten

                Thanks for the info. That's kind of what I suspected / figured, but didn't actually 100% know for sure. The whole thing came into question recently when my boss asked me to set up some reports for upper mgmt and asked me about the directionality. I just wanted to make sure I was telling him the correct answer. Didn't know the bit about Orion not using the data if both ports weren't being monitored, but it makes sense.

                 

                Thank you very much for the information!  I appreciate it.

                 

                John