Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

How to Use NPROBE / Netflow With Linux -- And a Question

Howdy,

First I want to thank Solarwinds for all their great work, the moderators and contributors here on Thwack for all their great questions and helpful answers, and the folks at Ntop for their really nice Netflow collector product "NProbe". Nprobe works really well and plays very nicely with Solarwinds NTA. But, as many folks (including myself) have discovered, it can be a royal pain to figure out and set up. I am not an expert at Netflow (nor NProbe) by any means, but I wanted a better way to configure and manage NProbe collections in our environment. For us it is an essential item of operation and as such merits a decent mechanism to control it.

First a little bit about our environment... In our setup we have a number of network "Zones" that we need to monitor and track usage stats for. We'll concentrate on just four of them in this discussion and we'll call them "DMZ", "USER", "SERVER" and "COREX", and assign them the (scrubbed) net-blocks of 2.2.2.0/24, 3.3.3.0/24, 4.4.4.0/24 and 5.5.5.0/24 respectively, and one additional network 1.2.3.4 for the collection server itself (which we call "MARS-NPROBE").

After purchasing the NProbe software from Ntop.org, compiling it and getting it set up on our system, I went through the usual routine of trying out all the options (including many iterations of the '-u' and '-Q' parameters) and finally got to the point where it seemed like it was pretty much working but I was still occasionally tweaking it a bit here and there, which involved a lot of starting, stopping, "ps"-ing and "grep"-ing over and over ad-naseum until I thought "there's got to be a better way"... what follows are the collection of setup and management scripts that I wrote for our organization. Perhaps they'll help you get started with NProbe and Netflow monitoring in yours...

It turns out that all those "configuration" options in "/etc/sysconfig/network-scripts" (e.g. "ifcfg-eth0", et al) are really nothing more than a collection of shell script variables that are set in a convenient, well-known place, that the actual interface control scripts ("ifup" & "ifdown") can use to properly configure the respective NIC port on the system. I kept thinking about those config files and thinking what a good place they make to put the Nprobe configuration if I could just figure out how to access them at the right time. My first attempt was a script that simply knew about the "/etc/sysconfig/network-scripts" location and read the various ifcfg files directly. Then I was poking around in the "ifup" control script and realized that it actually had a provision for local actions-- namely the "/sbin/ifup-local" and "/sbin/ifdown-local" scripts-- which I quickly realized would be the ideal mechanism for launching the Nprobe-specific control scripts. That way Nprobe would be started and/or stopped via the normal and customary action of starting or stopping the NIC port itself with no further intervention required. And being a rather lazy admin myself, that seemed like the perfect opportunity for an "elegant solution" AND a little less work on my part! A double win if there ever was one, right? :-)

Originally I was numbering my collection ports-- putting a numbered interface into each collection environment. Then it dawned on me that if the interface is promiscuous, it doesn't really need a number since it's going to see everything on the interface anyway. So you'll notice in the configuration scripts that the usual network configuration information for the collection ports is all commented-out. Also you will notice several new NProbe-specific options which are useful for passing information to the NProbe program-- such as a nice "NAME" for the NIC-- its helpful when you're looking through a "ps -ef | grep nprobe" output to be able to see which nprobe process is monitoring which zone. You can also pass information to describe the network to be monitored, etc. There are additional NProbe setup and configuration options which can be set in the NProbe control script itself, which lives in "/usr/local/sbin/nprobe-ctl.sh". I realize it's strictly not a stripped-binary, but sbin still seemed like an appropriate place to put it. You can put it wherever you like, just be sure to update the other scripts accordingly.

Over time, I have used several methods of communicating with the Solarwinds Orion NPM / NTA server. One very useful method, which I highly recommend if you have a spare NIC, is to use a dedicated NIC with a direct "crossover" style cable and just go port-to-port. If you do this, you only need a "private interface" set up between you and the other system-- so you can set the NIC ports to whatever private addresses you want, just be sure to make them on the same network. Presently, I am using the regular interface on the system and that works pretty well too. But it does put traffic back out on the switching network and so is technically disturbing the observation and contributing a little bit to the overhead on the network. I'll get around to reinstalling the crossover cable at some point, but I just wanted to point out that you can do it either way with acceptable results.

The other options in the file should, hopefully, be self-explanatory. And if not, it should only take a little sleuthing through the scripts to figure them out. But I do want to talk briefly about setting up the monitor sessions on the switches. We use Cisco gear so the monitor examples are specific to those, but the idea is similar and available on other types of switches. I've also included a sample "monitor session" configuration for a typical Cisco switch so you can see how that's done. The biggest thing to watch out for is that you don't include your monitor ("destination") port as one of the source ports. It may be that IOS won't actually let you do that so it may not be a big concern. For other switches you'll need to read the manual.

Now, here's my question-- it doesn't have anything to do with the "control" aspect of the Nprobe stuff, but rather back to those pesky '-u' and '-Q' NProbe options... I recently discovered that I only seem to be collecting INBOUND traffic-- at least from the perspective of Solarwinds NTA. There are only ingress stats available. Does this seem right? Or should I be seeing two streams-- an "Inbound" and an "Outbound" stream? If so, how do I achieve that?

Okay-- here are the scripts. Copy them to the locations indicated, adjust the settings to match your environment, and then they should just work. Don't forget to set the control scripts executable.

[mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth2

##--------------------------------------

## MARS-NPROBE-USER

##--------------------------------------

DEVICE=eth2

NAME="MARS-NPROBE-USER"

HWADDR=00:11:22:33:8B:D0

NM_CONTROLLED=no

ONBOOT=yes

TYPE=Ethernet

BOOTPROTO=none

IPV4_FAILURE_FATAL=no

IPV6INIT=no

USERCTL=no

DEFROUTE=no

## Basic Network Settings

#IPADDR=2.2.2.42

#NETWORK=2.2.2.0

#NETMASK=255.255.255.0

#GATEWAY=2.2.2.254

## Automagically fire-up NProbe

PROMISC=yes

NPROBE=yes

## Must be set for nprobe!

CIDR="2.2.2.0/24"

## Set this to ethNN

INDEX="2"

[mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth3

##--------------------------------------

## MARS-NPROBE-SERVER

##--------------------------------------

DEVICE=eth3

NAME="MARS-NPROBE-SERVER"

HWADDR=00:11:22:33:8B:D1

NM_CONTROLLED=no

ONBOOT=yes

TYPE=Ethernet

BOOTPROTO=none

IPV4_FAILURE_FATAL=no

IPV6INIT=no

USERCTL=no

DEFROUTE=no

## Basic Network Settings

#IPADDR=3.3.3.42

#NETWORK=3.3.3.0

#NETMASK=255.255.255.0

#GATEWAY=3.3.3.254

## Automagically fire-up NProbe

PROMISC=yes

NPROBE=yes

## Must be set for nprobe!

CIDR="3.3.3.0/24"

## Set this to ethNN

INDEX="3"

[mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth4

##--------------------------------------

## MARS-NPROBE-DMZ

##--------------------------------------

DEVICE=eth4

NAME="MARS-NPROBE-DMZ"

HWADDR=00:11:22:33:8B:D4

NM_CONTROLLED=no

ONBOOT=yes

TYPE=Ethernet

BOOTPROTO=none

IPV4_FAILURE_FATAL=no

IPV6INIT=no

USERCTL=no

DEFROUTE=no

## Basic Network Settings

#IPADDR=4.4.4.42

#NETWORK=4.4.4.0

#NETMASK=255.255.255.0

#GATEWAY=4.4.4.1

## Automagically fire-up NProbe

PROMISC=yes

NPROBE=yes

## Must be set for nprobe!

CIDR="4.4.4.0/24"

## Set this to ethNN

INDEX="4"

[mars-nprobe network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth5

##--------------------------------------

## MARS-NPROBE-COREX

##--------------------------------------

DEVICE=eth5

NAME="MARS-NPROBE-COREX"

HWADDR=00:11:22:33:8B:D5

NM_CONTROLLED=no

ONBOOT=yes

TYPE=Ethernet

BOOTPROTO=none

IPV4_FAILURE_FATAL=no

IPV6INIT=no

USERCTL=no

DEFROUTE=no

## Basic Network Settings

#IPADDR=5.5.5.42

#NETWORK=5.5.5.0

#NETMASK=255.255.255.0

#GATEWAY=5.5.5.1

## Automagically fire-up NProbe

PROMISC=yes

NPROBE=yes

## Must be set for nprobe!

CIDR="5.5.5.0/24"

## Set this to ethNN

INDEX="5"

[mars-nprobe sbin]# cat /sbin/ifup-local

#!/bin/sh

## ifup-local -- JWHITTEN -- 10/03/2011

## Script to handle local modifications to 'ifup' process

cd /etc/sysconfig/network-scripts

. ./network-functions

[ -f ../network ] && . ../network

unset REALDEVICE

if [ "$1" = --realdevice ] ; then

REALDEVICE=$2

shift 2

CONFIG=$1

need_config "${CONFIG}"

source_config

## see if we want PROMISC (Promiscuous) mode and turn it on

#echo "IFUP-LOCAL: CONFIG=${CONFIG}"

#echo "PROMISC WANTED? ${PROMISC}"

if [ "$PROMISC" = yes ]; then

# echo "COMMAND=/sbin/ifconfig ${DEVICE} promisc"

/sbin/ifconfig ${DEVICE} promisc

## see if we need to fire up an nprobe for this port

if [ "${NPROBE}" = yes ]; then

/usr/local/sbin/nprobe-ctl.sh start "${DEVICE}" "${NAME}"

[mars-nprobe sbin]# cat /sbin/ifdown-local

#!/bin/sh

## ifdown-local -- JWHITTEN -- 10/03/2011

## Script to handle local modifications to 'ifdown' process

cd /etc/sysconfig/network-scripts

. ./network-functions

[ -f ../network ] && . ../network

unset REALDEVICE

if [ "$1" = --realdevice ] ; then

REALDEVICE=$2

shift 2

CONFIG=$1

need_config "${CONFIG}"

source_config

## see if we want PROMISC (Promiscuous) mode and turn it off

#echo "IFDOWN-LOCAL: CONFIG=${CONFIG}"

#echo "PROMISC WANTED? ${PROMISC}"

if [ "$PROMISC" = yes ]; then

# echo "COMMAND=/sbin/ifconfig ${DEVICE} -promisc"

/sbin/ifconfig ${DEVICE} -promisc

## see if we need to shut down an nprobe for this port

if [ "${NPROBE}" = yes ]; then

/usr/local/sbin/nprobe-ctl.sh stop "${DEVICE}" "${NAME}"

[mars-nprobe sbin]# cat /usr/local/sbin/nprobe-ctl.sh

#! /bin/sh

#############################################################################

## RUN_NPROBES.SH -- revision 0.1 -- JWHITTEN -- 25-AUG-2011

## This is a first stab at a startup script for the Nprobe / NetFlow

## processing engine Nprobe processes SPAN/RSPAN traffic from switches

## and converts it into NetFlow traffic for the monitoring and security

## environments.

## At some point it will be necessary to revisit this script and add

## additional Nprobe instanaces to monitor additional incoming traffic

## (i.e., additional ethernet ports). Also the plan is to re-broadcast

## the Nprobe / NetFlow feed to OSSIM and Snort on direct, dedicated

## ethernet ports.

#############################################################################

##############################################################################

## Define some variables

##############################################################################

## Where PID files live (usually /var/run)

PID_DIR="/var/run"

## Where is the Nprobe binary located?

CMD_NPROBE="/usr/local/bin/nprobe"

## Which interface shall we monitor?

IFACE="eth0"

## MARS-ORION Dedicated Port

## Use with dedicated crossover cable

## COLLECTION_HOST="10.254.254.254"

## use if sent over the network

COLLECTION_HOST="1.2.3.4"

## Normal Port for NetFlow / NProbe

COLLECTION_PORT="2055"

## Beginning Offset into SNMP ID Table? (usually 1)

SNMP_IDX_START=1

## Ending Offset into SNMP ID Table? (perhaps 2? Not sure what this should be..??)

SNMP_IDX_END=2

## Prefix for Probe Instance Name

DEFAULT_INSTANCE_PREFIX="MARS-NPROBE"

## NProbe Single Opts

NPROBE_OPTS="-G -a"

## Where nprobe stats files live

STATS_DIR="/var/www/html/nprobe"

##============================================================================

## **** N O T I C E -- N O T I C E -- N O T I C E -- N O T I C E ****

## **** NO USER SERVICEABLE PARTS BELOW THIS LINE !!! ****

##============================================================================

##############################################################################

## Functions & Subroutines

##############################################################################

## load device config file

function need_device_config() {

CONFIG=${1}

cd /etc/sysconfig/network-scripts

. ./network-functions

[ -f ../network ] && . ../network

unset REALDEVICE

if [ "$1" = --realdevice ] ; then

REALDEVICE=$2

shift 2

need_config "${CONFIG}"

source_config

}

## construct an nprobe instance name

function need_instance_name() {

IFACE=${1}

## convert to upper case for display version

D_IFACE=`echo ${IFACE} | awk '{ print toupper($0) }'`

## How this instance will be referenced in the syslog

if [ "x${NAME}" = "x" ]; then

INSTANCE_NAME="${D_IFACE}-${DEFAULT_INSTANCE_PREFIX}"

else

INSTANCE_NAME="${D_IFACE}-${NAME}"

return

}

function need_bound_network() {

BOUND_NETWORK="${CIDR}@${INDEX}"

}

function need_pid_file_name() {

INSTANCE_NAME=${1}

PID_FILE="${PID_DIR}/${INSTANCE_NAME}.pid"

}

function need_stats_file_name() {

INSTANCE_NAME=${1}

STATS_FILE="${STATS_DIR}/${INSTANCE_NAME}.txt"

}

## log an action to the syslog

function log_action() {

ACTION=${1}

/usr/bin/logger "${ACTION}"

}

## start up an nprobe

function nprobe_start() {

## Our command line

IFACE=${1}

## see if we want a stats file

if [ "x${STATS_DIR}" != "x" ]; then

need_stats_file_name "${INSTANCE_NAME}"

OPT_STATS_FILE="-9 ${STATS_FILE}"

## see if we want to identify the network

NPROBE_PRIMARY_OPTS="${NPROBE_OPTS} -i ${IFACE} -n ${COLLECTION_HOST}:${COLLECTION_PORT} -I ${INSTANCE_NAME} -g ${PID_FILE}"

## bundle-up the snmp index opts

# OPT_SNMP_IDX="-u ${SNMP_IDX_START} -Q ${SNMP_IDX_END}"

OPT_SNMP_IDX="-u ${INDEX} -Q ${INDEX}"

## set up network binding for port/network identification

need_bound_network

OPT_BOUND_NETWORK="-L ${BOUND_NETWORK}"

## assemble nprobe command

CMD="${CMD_NPROBE} ${NPROBE_PRIMARY_OPTS} ${OPT_STATS_FILE} ${OPT_SNMP_IDX} ${OPT_BOUND_NETWORK}"

echo "START COMMAND=${CMD}"

## execute the command

${CMD} &

echo "Starting NProbe ${INSTANCE_NAME}"

log_action "${CMD}"

log_action "NProbe Instance ${INSTANCE_NAME} Started"

}

## shutdown an nprobe

function nprobe_stop() {

IFACE=${1}

# echo $"Stopping ${INSTANCE_NAME}... (PID_FILE=${PID_FILE}) "

MSG="Stopping NProbe ${INSTANCE_NAME} "

## if it exists, see if we can kill it

if [ -f "${PID_FILE}" ]; then

kill 15 `cat ${PID_FILE}` && {

echo "${MSG} SUCCESS"

log_action "${MSG} SUCCESS"

} || {

echo "${MSG} FAILED"

log_action "${MSG} FAILED"

}

# else

# echo $"Pid File for NProbe Instance ${INSTANCE_NAME} Not Found" >&2

}

function display_help() {

## doesn't return

echo $"Usage: nprobe-ctl.sh [start|stop] <device name> {<nickname>} " >&2

exit 1

}

##############################################################################

## Begin Script

##############################################################################

COMMAND=${1}; shift

IFACE=${1}; shift

[ -z "${COMMAND}" -o -z "${IFACE}" ] && {

display_help

}

#echo "COMMAND=${COMMAND}, IFACE=${IFACE}"

need_device_config "${IFACE}"

need_instance_name "${IFACE}" "${NAME}"

need_pid_file_name "${INSTANCE_NAME}"

case "${COMMAND}" in

start | START)

nprobe_start "${IFACE}"

;;

stop | STOP)

nprobe_stop "${IFACE}"

;;

display_help

;;

esac

This is what it looks like when its running...

[mars-nprobe sbin]# ps -ef | grep nprobe

nobody 29352 1 1 Jul23 ? 00:19:53 /usr/local/bin/nprobe -G -a -i eth2 -n 1.2.3.4:2055 -I ETH2-MARS-NPROBE-USER -g /var/run/ETH2-MARS-NPROBE-USER.pid -9 /var/www/html/nprobe/ETH2-MARS-NPROBE-USER.txt -u 2 -Q 2 -L 2.2.2.0/24@2

nobody 29488 1 6 Jul23 ? 01:01:41 /usr/local/bin/nprobe -G -a -i eth3 -n 1.2.3.4:2055 -I ETH3-MARS-NPROBE-SERVER -g /var/run/ETH3-MARS-NPROBE-SERVER.pid -9 /var/www/html/nprobe/ETH3-MARS-NPROBE-SERVER.txt -u 3 -Q 3 -L 3.3.3.0/24@3

nobody 29624 1 2 Jul23 ? 00:22:51 /usr/local/bin/nprobe -G -a -i eth4 -n 1.2.3.4:2055 -I ETH4-MARS-NPROBE-DMZ -g /var/run/ETH4-MARS-NPROBE-DMZ.pid -9 /var/www/html/nprobe/ETH4-MARS-NPROBE-DMZ.txt -u 4 -Q 4 -L 4.4.4.0/24@4

nobody 29759 1 0 Jul23 ? 00:03:56 /usr/local/bin/nprobe -G -a -i eth5 -n 1.2.3.4:2055 -I ETH5-MARS-NPROBE-COREX -g /var/run/ETH5-MARS-NPROBE-COREX.pid -9 /var/www/html/nprobe/ETH5-MARS-NPROBE-COREX.txt -u 5 -Q 5 -L 5.5.5.0/24@5

(On Cisco 3750 or 2960 Switch, the 'monitor session' portion from a 'show run'. Obviously you don't want to include the destination port in the source monitor.):

monitor session 42 source interface Gi1/0/1 - 19 , Gi1/0/21 , Gi1/0/23 - 24

monitor session 42 destination interface Gi1/0/22 encapsulation replicate

Find more posts tagged with

Accepted answers

Martin.Krivanek

nProbe takes two parameters (-u and -Q) to specify input and output interface index in flows it generates. For example -u 1 -Q 2 means that all flows will have 1 as input if index and 2 as output if index. From NTA perspective then all flows appear as ingress on first interface and egress on the other. Both interfaces on the machine where nProbe runs (index 1 and 2 in this example) have to be in Orion. If only one interface is polled by Orion, and I think this is the case, NTA won’t resolve if index 2 and all flows will only appear as ingress on first interface.

But this doesn’t matter much, since information about original interfaces is lost with nProbe anyway. You would see exactly same traffic on both interfaces, it would just be ingress on one and egress on the other. This is only about viewing data, nothing is lost. So nProbe provides both direction, it’s just that flow direction is kind of useless with nProbe – all traffic appears as ingress on one interface and optionally as egress on the other.

Let me know if you need more information.

Regards,

Martin

All comments

mavturner

Thanks for the great write up and information!

I believe nProbe will also show as Ingress because that is how it sees the traffic based on its position in the network. I've asked the engineering team here to confirm.

Martin.Krivanek

Let me know if you need more information.

Regards,

Martin

jwhitten

Thanks for the info. That's kind of what I suspected / figured, but didn't actually 100% know for sure. The whole thing came into question recently when my boss asked me to set up some reports for upper mgmt and asked me about the directionality. I just wanted to make sure I was telling him the correct answer. Didn't know the bit about Orion not using the data if both ports weren't being monitored, but it makes sense.

Thank you very much for the information! I appreciate it.

John