In /etc/flow-tools/cfg/stat.cfg the default report is:

stat-report default
  type @{TYPE:-summary-detail}
  output
    format ascii
    sort @{SORT:-+}
    fields @{FIELDS:-+}
    options @{OPTIONS:-+header,+xheader,+totals}
    path |flow-rptfmt @{RPTOPT:--f ascii}

stat-definition default
  report default

This is a fairly generic report. But notice that many of the options can be overridden. For example, type @{TYPE:-summary-detail} means that the default is “summary-detail”, but can be overridden on the command line.

The following will produce a report of all the hosts a particular IP talked to:

 flow-cat ft* | flow-nfilter -F ip-src-addr -v ADDR=x.x.x.x |  flow-report -v TYPE=ip-source/destination-address

• flow-cat ft* – displays all the netflow files in the directory
• flow-nfilter -F ip-src-addr -v ADDR=x.x.x.x – filter out everything except the source x.x.x.x
• flow-report -v TYPE=ip-source/destination-address – generates a ip-source/destination-address report (since we’ve only got one IP as an input to this, an ip-destination-address report might have worked just as well

Another one which I’ve just used:

flow-cat ft* | flow-nfilter -F ip-src-net -v ADDR=x.x.x.0/24 |  flow-report -v TYPE=ip-source-address -v SORT=+octets

• flow-cat ft* – displays all the netflow files in the directory
• flow-nfilter -F ip-src-net -v ADDR=x.x.x.0/24 – filter out everything except stuff coming from x.x.x.0/24 (this filter was created in the previous post)
• flow-report -v TYPE=ip-source-address -v SORT=+octets – summarize on source address, and sort by octets to give the top talkers in the subnet

Content Copyright Sean Walberg

More command line NetFlow reports

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The next useful thing would be to pull out all flows to or from a particular network. To do so, we’ll have to define a new primitive that is a variable network/netmask, and then a filter specifying a match on stuff to or from that netmask.

flow-nfilter provides two primitives, the ip-address-mask and ip-address-prefix. The mask expects a traditional netmask (255.255.255.0), while the prefix expects CIDR notation (/24). I like the latter.

The primitive is copied from others:

filter-primitive VAR_PREFIX
  type ip-address-prefix
  permit @{ADDR}

The filter is then

filter-definition ip-addr
  match ip-destination-address VAR_PREFIX
        or
  match ip-source-address VAR_PREFIX

The "or" keyword means that either condition can be matched.

This filter will let you

The flow-report tool reads in a report definition from /etc/flow-tools/cfg/stat.cfg, and sends out a summary of the results. This tool can also summarize based on time. Here's a report of all the conversations per 10 minute interval, sorted by number of octets:

stat-report talkers
  type ip-source/destination-address
  output
    sort +octets
    fields +other
    path /tmp/talkers
stat-definition talkers
  time-series 600
  report talkers

The command line is then

flow-cat /var/flow-tools/ft-v05.2008-12-28.* | flow-nfilter -F ip-addr -v ADDR=172.16.128.0/17 | flow-report -S talkers

This will summarize the conversations for traffic to or from the 172.16.128.0/17 network into /tmp/talkers. A report like this could be fed into another script to generate data to be graphed, or inspected by hand to find out who was talking during the desired period.

Content Copyright Sean Walberg

Netflow – Filtering on network and producing a report

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

First, I should mention that there may be easier ways of doing this. The flow-tools package includes a lot of tools, and I feel like I learn something new each time I use them. I also tend to use a lot of shell scripting, so I may do this a different way each time.

Going back to the idea of a flow, I should be able to figure out the connection rate by looking at the number of inbound flows per second. A flow is half of a conversation (see the end of an article for an exception).

flow-cat can take the name of a file (or files), or the name of a directory, in which case it spits out all the files in the directory. On my Internet collector, I pass “-N -1″ to flow-capture to have the flows in separate directories per day (on my internal collector I don’t, go figure).

The first thing to do is filter my flows so that only incoming connections to the web server are caught. The flow-nfilter command can do this.

/etc/flow-tools/cfg/filter.cfg contains the filters. Some predefined ones are there for you, most notably, a “filter by destination address”:

filter-definition ip-dst-addr
  match ip-destination-address VAR_ADDR

This defines a filter that matches a destination address of VAR_ADDR. But what's VAR_ADDR? Earlier on in the file you'll see:

filter-primitive VAR_ADDR
  type ip-address
  permit @{ADDR:-0.0.0.0}

This primitive is an ip address, and either takes the value of ADDR, or failing that, 0.0.0.0.

Looking through the flow-nfilter manpage, you can set variables on the command line with -v. So, to see only the incoming flows to the web server, you get

flow-cat /var/flow-tools/2008-12-22/ | flow-nfilter -F ip-dst-addr -v  ADDR=x.x.x.x | flow-print

This calls the ip-dst-addr filter and assigns x.x.x.x (the server address) to the ADDR field.

With a bit of shell magic, you can iterate through all the files in the directory and use the filename to write out the time, and then count the number of flows in the file:

for i in /var/flow-tools/2008-12-22/ft*; do echo -n $i| sed 's/.*\.\(..\)\(..\).*/\1:\2/' ; echo -n " ";  flow-cat $i| flow-nfilter  -F ip-dst-addr -v  ADDR=x.x.x.x| wc -l; done > data

I've redirected the output to a file called "data", which looks like this:

[root@netflow ~]# head data
00:00 9388
00:05 17850
00:10 15280
00:15 11759
00:20 14840
00:25 11018
...

The last step is to use Gnuplot to plot the data. Start by typing "gnuplot"

set timefmt "%H:%M"
set xdata time
set terminal png large color picsize 1200 480
set output '/var/www/html/stats.png'
plot 'data' using 1:($2/300) with linespoints

I then look at stats.png through my web browser. In this case, I went back and edited the data file to cut down on the number of datapoints around the outage, which ends up with something like:

From the graph I can see a few places where the connection rate drops which is indicative of a problem. However after that, the website is able to keep up.

When is a flow not a flow?

I had the command

ip flow-cache timeout active 2

in the configuration that I use. This sets the timeout of an active flow to 2 minutes. If a file transfer goes for longer than 2 minutes, the router will stop that flow and create a new one.

Normally, if you're using 5 minute data and you have a transfer that takes 6 minutes, the flow record will be written when the flow expires. All the transfer will look like it happened in the 6th minute, which really skews your stats. Breaking the flow up into 3 smaller flows, each about 2 minutes long, makes the effect less noticeable.

The flow start/stop times are always written to the flow, but often we're doing simpler analysis of the flows and don't take the time to resort the data (there's going to be a lot of it, after all!).

Content Copyright Sean Walberg

Netflow – a simple filter and getting connection stats

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

First, set up your router to export flow information:

ip flow-cache timeout active 2
mls flow ip full
mls flow ipx destination
mls nde sender
mls nde interface
mls nde flow include protocol tcp
ip flow-export source GigabitEthernet1/1
ip flow-export version 5 origin-as
ip flow-export destination X.X.X.X 2055

Where X.X.X.X is the address of your NetFlow collector, and GigabitEthernet1/1 is the router's interface on that subnet. (This was taken from a 7600 router, you may not need the NDE stuff if you're on a different platform)

Then, on each interface you want to capture flows for,

ip route-cache flow

You can check on the status of the export with

ROUTER#show ip flow export
Flow export is enabled
  Exporting flows to X.X.X.X (2055)
  Exporting using source interface GigabitEthernet1/1
  Version 5 flow records, origin-as
  235556663 flows exported in 7945727 udp datagrams
  0 flows failed due to lack of export packet
  743 export packets were sent up to process level
  0 export packets were dropped due to no fib
  18425 export packets were dropped due to adjacency issues
  0 export packets were dropped due to fragmentation failures
  0 export packets were dropped due to encapsulation fixup failures
  0 export packets were dropped enqueuing for the RP
  0 export packets were dropped due to IPC rate limiting

You can immediately see some statistics now that you have NetFlow enabled:

#show ip cache flow
IP packet size distribution (4086M total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .001 .627 .032 .012 .020 .019 .085 .009 .001 .002 .003 .005 .006 .006 .006
    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .005 .004 .005 .066 .079 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 4456704 bytes
  417 active, 65119 inactive, 235561367 added
  132171494 ager polls, 0 flow alloc failures
  Active flows timeout in 2 minutes
  Inactive flows timeout in 15 seconds
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet       12352      0.0        24    44      0.0       3.9      13.3
TCP-FTP          50507      0.0         1    55      0.0       0.7      14.4
TCP-FTPD         18867      0.0         1   499      0.0       0.5      15.0
TCP-WWW      158177053     36.8        17   186    627.8       3.3       8.9
TCP-SMTP        139330      0.0         1   135      0.0       0.0      15.4
TCP-X               23      0.0         2   222      0.0       1.8       9.4
TCP-BGP              2      0.0         1    64      0.0       0.0      15.7
TCP-NNTP             3      0.0         1    56      0.0       0.0      11.0
TCP-other     17276962      4.0        21   318     85.9       3.1       8.8
UDP-DNS        2866156      0.6         1    68      0.8       0.5      15.4
UDP-NTP        2082119      0.4         1    84      0.4       0.0      15.4
UDP-TFTP           137      0.0         5    49      0.0      20.4      15.5
UDP-Frag          3796      0.0     26195  1394     23.1      20.5      14.7
UDP-other     48352973     11.2        15   275    173.6      10.8      14.8
ICMP           3302490      0.7         6   165      5.0       6.5      14.8
GRE            1844456      0.4        38   137     16.7     116.5       1.1
IP-other       1433724      0.3        53    52     17.8     111.4       2.5
Total:       235560950     54.8        17   240    951.5       6.4      10.3

To collect the flows, install the flow-tools package, with either

yum install flow-tools

or whatever your distribution uses (

apt-get install flow-tools

), or install from source.

The flow-capture utility is the one that is used to write the flows to disk. It must be configured with the port (2055 in our case), and a location to write the flows to. In CentOS/RedHat/Fedora, this is done in /etc/sysconfig/flow-capture.

OPTIONS="-n 287 -N 0 -w /var/flow-tools -S 5 0/0/2055"

• -n 287: 287 files per day, or one file every 5 minutes. I recommend doing this instead of the default 15 minutes so that you have more real time access to your data, and some tools depend on this reporting interval.
• -N 0: Don't nest the files. All the flow files will be in one directory instead of one per day.
• -w /var/flow-tools: Write to this directory
• -S 5: Syslog a message every 5 minutes with the collection statistics
• 0/0/2055: listen on all interfaces to all exporters on port 2055

You may also want to configure something like tmpwatch in cron to clean up files (/usr/sbin/tmpwatch 720 /var/flow-tools) to only keep the last month or whatever you want. On a pipe that's used 100-200MB/sec, you can expect at least 10G of data to be logged.

Start flow-capture (service flow-capture start), and look for files in /var/flow-tools.

The files are binary, so you can't look at them directly. To have a look at what's there:

# flow-cat /var/flow-tools/ft-v05.2008-12-22.080500-0600 | flow-print | head
srcIP            dstIP            prot  srcPort  dstPort  octets      packets
x.x.x.105      x.x.x.151        6     4511     80       744         6
x.x.x.105      x.x.x.151        6     4512     80       985         12
x.x.x.105      x.x.x.151        6     4514     80       784         7
x.x.x.105      x.x.x.185        6     4516     80       985         6
x.x.x.105      x.x.x.52         6     4517     80       1744        7
x.x.x.105      x.x.x.41         6     4518     80       850         5
x.x.x.115      x.x.x.255       17    138      138      229         1
x.x.x.252      x.x.x.62         6     2727     80       40          1
x.x.x.105      x.x.x.27         6     4521     80       2221        22

The fields should be fairly self explanatory. The -f parameter to flow-print allows you to print out new data.

Coming up... Quick and dirty ways to report on your data.

Content Copyright Sean Walberg

Collecting NetFlow data

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

NetFlow is concerned with flows, which are a one way session between a source and a destination. The router is already caching information about the flow to help with the routing/switching function, NetFlow is an export of this information.

If you SSH to a server, that generates two flows. One is the connection from your ephemeral port to port 22 of the server, and one from port 22 back to your ephemeral port.

The analysis available with NetFlow is more fine-grained than what you get with SNMP. The flow contains the start and end time of the flow, the source and destination IP addresses and ports, the amount of data transferred, and autonomous system (AS) information (if the router is running BGP). There are other things, such as TCP flag information, QoS tags, and optional proprietary information, but the above gives us enough to proceed.

I’ve been playing with NetFlow for a while and have generated various reports. Every time I do something I seem to be starting from scratch, so I’m going to formalize my work on this blog. At the moment I am working on two NetFlow related projects. The first is to figure out the breakdown of protocols over our WAN. The second is to analyze our Internet usage, analyze peering, and detect DDOS traffic patterns in near-real time, or on an ad-hoc basis. I use the flow-tools package for Linux, along with some shell/perl/ruby scripting.

The next post will be about setting up the environment to capture flows on the router and the Linux box.

Content Copyright Sean Walberg

Introduction to NetFlow

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.