BSCI becomes ROUTE
BCMSN becomes SWITCH
ISCW and ONT become TSHOOT

Looks like more IPv6 content, multicast and IS-IS are gone, and more focus on simulations and troubleshooting.

The old exams are phased out on July 31, so if you have can make it by that date, get cracking!

Exams are now $200.

Content Copyright Sean Walberg

CCNP completely redone

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

In /etc/flow-tools/cfg/stat.cfg the default report is:

stat-report default
  type @{TYPE:-summary-detail}
  output
    format ascii
    sort @{SORT:-+}
    fields @{FIELDS:-+}
    options @{OPTIONS:-+header,+xheader,+totals}
    path |flow-rptfmt @{RPTOPT:--f ascii}

stat-definition default
  report default

This is a fairly generic report. But notice that many of the options can be overridden. For example, type @{TYPE:-summary-detail} means that the default is “summary-detail”, but can be overridden on the command line.

The following will produce a report of all the hosts a particular IP talked to:

 flow-cat ft* | flow-nfilter -F ip-src-addr -v ADDR=x.x.x.x |  flow-report -v TYPE=ip-source/destination-address

• flow-cat ft* – displays all the netflow files in the directory
• flow-nfilter -F ip-src-addr -v ADDR=x.x.x.x – filter out everything except the source x.x.x.x
• flow-report -v TYPE=ip-source/destination-address – generates a ip-source/destination-address report (since we’ve only got one IP as an input to this, an ip-destination-address report might have worked just as well

Another one which I’ve just used:

flow-cat ft* | flow-nfilter -F ip-src-net -v ADDR=x.x.x.0/24 |  flow-report -v TYPE=ip-source-address -v SORT=+octets

• flow-cat ft* – displays all the netflow files in the directory
• flow-nfilter -F ip-src-net -v ADDR=x.x.x.0/24 – filter out everything except stuff coming from x.x.x.0/24 (this filter was created in the previous post)
• flow-report -v TYPE=ip-source-address -v SORT=+octets – summarize on source address, and sort by octets to give the top talkers in the subnet

Content Copyright Sean Walberg

More command line NetFlow reports

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Router> enable
Router# conf t
Router# access-list 100 permit tcp any any eq ?

This will result in the following table:

 <0-65535>    Port number
bgp          Border Gateway Protocol (179)
chargen      Character generator (19)
cmd          Remote commands (rcmd, 514)
daytime      Daytime (13)
discard      Discard (9)
domain       Domain Name Service (53)
echo         Echo (7)
exec         Exec (rsh, 512)
finger       Finger (79)
ftp          File Transfer Protocol (21)
ftp-data     FTP data connections (used infrequently, 20)
gopher       Gopher (70)
hostname     NIC hostname server (101)
ident        Ident Protocol (113)
irc          Internet Relay Chat (194)
klogin       Kerberos login (543)
kshell       Kerberos shell (544)
login        Login (rlogin, 513)
lpd          Printer service (515)
nntp         Network News Transport Protocol (119)
pim-auto-rp  PIM Auto-RP (496)
pop2         Post Office Protocol v2 (109)
pop3         Post Office Protocol v3 (110)
smtp         Simple Mail Transport Protocol (25)
sunrpc       Sun Remote Procedure Call (111)
syslog       Syslog (514)
tacacs       TAC Access Control System (49)
talk         Talk (517)
telnet       Telnet (23)
time         Time (37)
uucp         Unix-to-Unix Copy Program (540)
whois        Nicname (43)
www          World Wide Web (HTTP, 80) 

Content Copyright Sean Walberg

Checking common port numbers on Cisco IOS

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The next useful thing would be to pull out all flows to or from a particular network. To do so, we’ll have to define a new primitive that is a variable network/netmask, and then a filter specifying a match on stuff to or from that netmask.

flow-nfilter provides two primitives, the ip-address-mask and ip-address-prefix. The mask expects a traditional netmask (255.255.255.0), while the prefix expects CIDR notation (/24). I like the latter.

The primitive is copied from others:

filter-primitive VAR_PREFIX
  type ip-address-prefix
  permit @{ADDR}

The filter is then

filter-definition ip-addr
  match ip-destination-address VAR_PREFIX
        or
  match ip-source-address VAR_PREFIX

The "or" keyword means that either condition can be matched.

This filter will let you

The flow-report tool reads in a report definition from /etc/flow-tools/cfg/stat.cfg, and sends out a summary of the results. This tool can also summarize based on time. Here's a report of all the conversations per 10 minute interval, sorted by number of octets:

stat-report talkers
  type ip-source/destination-address
  output
    sort +octets
    fields +other
    path /tmp/talkers
stat-definition talkers
  time-series 600
  report talkers

The command line is then

flow-cat /var/flow-tools/ft-v05.2008-12-28.* | flow-nfilter -F ip-addr -v ADDR=172.16.128.0/17 | flow-report -S talkers

This will summarize the conversations for traffic to or from the 172.16.128.0/17 network into /tmp/talkers. A report like this could be fed into another script to generate data to be graphed, or inspected by hand to find out who was talking during the desired period.

Content Copyright Sean Walberg

Netflow – Filtering on network and producing a report

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

First, I should mention that there may be easier ways of doing this. The flow-tools package includes a lot of tools, and I feel like I learn something new each time I use them. I also tend to use a lot of shell scripting, so I may do this a different way each time.

Going back to the idea of a flow, I should be able to figure out the connection rate by looking at the number of inbound flows per second. A flow is half of a conversation (see the end of an article for an exception).

flow-cat can take the name of a file (or files), or the name of a directory, in which case it spits out all the files in the directory. On my Internet collector, I pass “-N -1″ to flow-capture to have the flows in separate directories per day (on my internal collector I don’t, go figure).

The first thing to do is filter my flows so that only incoming connections to the web server are caught. The flow-nfilter command can do this.

/etc/flow-tools/cfg/filter.cfg contains the filters. Some predefined ones are there for you, most notably, a “filter by destination address”:

filter-definition ip-dst-addr
  match ip-destination-address VAR_ADDR

This defines a filter that matches a destination address of VAR_ADDR. But what's VAR_ADDR? Earlier on in the file you'll see:

filter-primitive VAR_ADDR
  type ip-address
  permit @{ADDR:-0.0.0.0}

This primitive is an ip address, and either takes the value of ADDR, or failing that, 0.0.0.0.

Looking through the flow-nfilter manpage, you can set variables on the command line with -v. So, to see only the incoming flows to the web server, you get

flow-cat /var/flow-tools/2008-12-22/ | flow-nfilter -F ip-dst-addr -v  ADDR=x.x.x.x | flow-print

This calls the ip-dst-addr filter and assigns x.x.x.x (the server address) to the ADDR field.

With a bit of shell magic, you can iterate through all the files in the directory and use the filename to write out the time, and then count the number of flows in the file:

for i in /var/flow-tools/2008-12-22/ft*; do echo -n $i| sed 's/.*\.\(..\)\(..\).*/\1:\2/' ; echo -n " ";  flow-cat $i| flow-nfilter  -F ip-dst-addr -v  ADDR=x.x.x.x| wc -l; done > data

I've redirected the output to a file called "data", which looks like this:

[root@netflow ~]# head data
00:00 9388
00:05 17850
00:10 15280
00:15 11759
00:20 14840
00:25 11018
...

The last step is to use Gnuplot to plot the data. Start by typing "gnuplot"

set timefmt "%H:%M"
set xdata time
set terminal png large color picsize 1200 480
set output '/var/www/html/stats.png'
plot 'data' using 1:($2/300) with linespoints

I then look at stats.png through my web browser. In this case, I went back and edited the data file to cut down on the number of datapoints around the outage, which ends up with something like:

From the graph I can see a few places where the connection rate drops which is indicative of a problem. However after that, the website is able to keep up.

When is a flow not a flow?

I had the command

ip flow-cache timeout active 2

in the configuration that I use. This sets the timeout of an active flow to 2 minutes. If a file transfer goes for longer than 2 minutes, the router will stop that flow and create a new one.

Normally, if you're using 5 minute data and you have a transfer that takes 6 minutes, the flow record will be written when the flow expires. All the transfer will look like it happened in the 6th minute, which really skews your stats. Breaking the flow up into 3 smaller flows, each about 2 minutes long, makes the effect less noticeable.

The flow start/stop times are always written to the flow, but often we're doing simpler analysis of the flows and don't take the time to resort the data (there's going to be a lot of it, after all!).

Content Copyright Sean Walberg

Netflow – a simple filter and getting connection stats

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

First, set up your router to export flow information:

ip flow-cache timeout active 2
mls flow ip full
mls flow ipx destination
mls nde sender
mls nde interface
mls nde flow include protocol tcp
ip flow-export source GigabitEthernet1/1
ip flow-export version 5 origin-as
ip flow-export destination X.X.X.X 2055

Where X.X.X.X is the address of your NetFlow collector, and GigabitEthernet1/1 is the router's interface on that subnet. (This was taken from a 7600 router, you may not need the NDE stuff if you're on a different platform)

Then, on each interface you want to capture flows for,

ip route-cache flow

You can check on the status of the export with

ROUTER#show ip flow export
Flow export is enabled
  Exporting flows to X.X.X.X (2055)
  Exporting using source interface GigabitEthernet1/1
  Version 5 flow records, origin-as
  235556663 flows exported in 7945727 udp datagrams
  0 flows failed due to lack of export packet
  743 export packets were sent up to process level
  0 export packets were dropped due to no fib
  18425 export packets were dropped due to adjacency issues
  0 export packets were dropped due to fragmentation failures
  0 export packets were dropped due to encapsulation fixup failures
  0 export packets were dropped enqueuing for the RP
  0 export packets were dropped due to IPC rate limiting

You can immediately see some statistics now that you have NetFlow enabled:

#show ip cache flow
IP packet size distribution (4086M total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .001 .627 .032 .012 .020 .019 .085 .009 .001 .002 .003 .005 .006 .006 .006
    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .005 .004 .005 .066 .079 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 4456704 bytes
  417 active, 65119 inactive, 235561367 added
  132171494 ager polls, 0 flow alloc failures
  Active flows timeout in 2 minutes
  Inactive flows timeout in 15 seconds
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet       12352      0.0        24    44      0.0       3.9      13.3
TCP-FTP          50507      0.0         1    55      0.0       0.7      14.4
TCP-FTPD         18867      0.0         1   499      0.0       0.5      15.0
TCP-WWW      158177053     36.8        17   186    627.8       3.3       8.9
TCP-SMTP        139330      0.0         1   135      0.0       0.0      15.4
TCP-X               23      0.0         2   222      0.0       1.8       9.4
TCP-BGP              2      0.0         1    64      0.0       0.0      15.7
TCP-NNTP             3      0.0         1    56      0.0       0.0      11.0
TCP-other     17276962      4.0        21   318     85.9       3.1       8.8
UDP-DNS        2866156      0.6         1    68      0.8       0.5      15.4
UDP-NTP        2082119      0.4         1    84      0.4       0.0      15.4
UDP-TFTP           137      0.0         5    49      0.0      20.4      15.5
UDP-Frag          3796      0.0     26195  1394     23.1      20.5      14.7
UDP-other     48352973     11.2        15   275    173.6      10.8      14.8
ICMP           3302490      0.7         6   165      5.0       6.5      14.8
GRE            1844456      0.4        38   137     16.7     116.5       1.1
IP-other       1433724      0.3        53    52     17.8     111.4       2.5
Total:       235560950     54.8        17   240    951.5       6.4      10.3

To collect the flows, install the flow-tools package, with either

yum install flow-tools

or whatever your distribution uses (

apt-get install flow-tools

), or install from source.

The flow-capture utility is the one that is used to write the flows to disk. It must be configured with the port (2055 in our case), and a location to write the flows to. In CentOS/RedHat/Fedora, this is done in /etc/sysconfig/flow-capture.

OPTIONS="-n 287 -N 0 -w /var/flow-tools -S 5 0/0/2055"

• -n 287: 287 files per day, or one file every 5 minutes. I recommend doing this instead of the default 15 minutes so that you have more real time access to your data, and some tools depend on this reporting interval.
• -N 0: Don't nest the files. All the flow files will be in one directory instead of one per day.
• -w /var/flow-tools: Write to this directory
• -S 5: Syslog a message every 5 minutes with the collection statistics
• 0/0/2055: listen on all interfaces to all exporters on port 2055

You may also want to configure something like tmpwatch in cron to clean up files (/usr/sbin/tmpwatch 720 /var/flow-tools) to only keep the last month or whatever you want. On a pipe that's used 100-200MB/sec, you can expect at least 10G of data to be logged.

Start flow-capture (service flow-capture start), and look for files in /var/flow-tools.

The files are binary, so you can't look at them directly. To have a look at what's there:

# flow-cat /var/flow-tools/ft-v05.2008-12-22.080500-0600 | flow-print | head
srcIP            dstIP            prot  srcPort  dstPort  octets      packets
x.x.x.105      x.x.x.151        6     4511     80       744         6
x.x.x.105      x.x.x.151        6     4512     80       985         12
x.x.x.105      x.x.x.151        6     4514     80       784         7
x.x.x.105      x.x.x.185        6     4516     80       985         6
x.x.x.105      x.x.x.52         6     4517     80       1744        7
x.x.x.105      x.x.x.41         6     4518     80       850         5
x.x.x.115      x.x.x.255       17    138      138      229         1
x.x.x.252      x.x.x.62         6     2727     80       40          1
x.x.x.105      x.x.x.27         6     4521     80       2221        22

The fields should be fairly self explanatory. The -f parameter to flow-print allows you to print out new data.

Coming up... Quick and dirty ways to report on your data.

Content Copyright Sean Walberg

Collecting NetFlow data

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

NetFlow is concerned with flows, which are a one way session between a source and a destination. The router is already caching information about the flow to help with the routing/switching function, NetFlow is an export of this information.

If you SSH to a server, that generates two flows. One is the connection from your ephemeral port to port 22 of the server, and one from port 22 back to your ephemeral port.

The analysis available with NetFlow is more fine-grained than what you get with SNMP. The flow contains the start and end time of the flow, the source and destination IP addresses and ports, the amount of data transferred, and autonomous system (AS) information (if the router is running BGP). There are other things, such as TCP flag information, QoS tags, and optional proprietary information, but the above gives us enough to proceed.

I’ve been playing with NetFlow for a while and have generated various reports. Every time I do something I seem to be starting from scratch, so I’m going to formalize my work on this blog. At the moment I am working on two NetFlow related projects. The first is to figure out the breakdown of protocols over our WAN. The second is to analyze our Internet usage, analyze peering, and detect DDOS traffic patterns in near-real time, or on an ad-hoc basis. I use the flow-tools package for Linux, along with some shell/perl/ruby scripting.

The next post will be about setting up the environment to capture flows on the router and the Linux box.

Content Copyright Sean Walberg

Introduction to NetFlow

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Take, for example, the following

 ! default route
 ip route 0.0.0.0 0.0.0.0 1.1.1.1
 ! pick your routing protocol
 router XXXXX
    redistribute static

Under what conditions will the default route make it into the routing protocol? The docs seem to indicate that the process is automatic in EIGRP, but requires intervention in IS-IS, OSPF, and BGP. Let's validate.

Take a simple network:

1.1.1.0/24 [R1] 2.2.2.0/24 [R2] 3.3.3.0/24 [R3]

(dynagen .net)

On R1, I have

router eigrp 1
 redistribute static
 network 1.0.0.0
 network 2.0.0.0
 no auto-summary
!
ip route 0.0.0.0 0.0.0.0 1.1.1.99

and over on R3

D*EX 0.0.0.0/0 [170/33280] via 3.3.3.1, 00:01:00, FastEthernet0/0

One thing to note is that the network statement specified 1.1.1.0 and 2.2.2.0, and not 0.0.0.0. This is because the network statement in the EIGRP config is used to match the interfaces that EIGRP will run on, which is where the network information comes from. network 0.0.0.0 would match both interfaces, but still would not add the default route. The redistribute static is what causes the route to get into EIGRP (which is why it shows up in R3's routing table as D*EX)

In OSPF, we have on R1

router ospf 1
 log-adjacency-changes
 redistribute static subnets
 network 0.0.0.0 255.255.255.255 area 0

This time I used network 0.0.0.0 to save some typing. However R3 does not see the default route. It does see another static route I put in to 9.9.9.0/24, so we know redistribution works properly.

The solution here is the default-information originate command. Adding "default-information originate" to the OSPF config solves this:

O*E2 0.0.0.0/0 [110/1] via 3.3.3.1, 00:00:02, FastEthernet0/0

For BGP, R1 is set up as follows:

R1#show run | section router bgp
router bgp 1
 no synchronization
 bgp log-neighbor-changes
 network 1.1.1.0
 network 2.2.2.0
 neighbor 2.2.2.2 remote-as 2
 no auto-summary

If I redistribute static, the default route does not show up on R2:

R2#show ip route bgp
     9.0.0.0/24 is subnetted, 1 subnets
B       9.9.9.0 [20/0] via 2.2.2.1, 00:00:31

At least two options exist. 1 is to do the "default-information originate" which allows the static route to be redistributed into BGP. The other is to not redistribute, but use the network 0.0.0.0 command to advertise the default route. In BGP, the network statement specifies the routes to be advertised, and not the interfaces like OSPF and EIGRP.

The difference between the two options, though, is in the BGP attributes.

! redistribute static and default-information originate
R2#show ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 8
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  1
    2.2.2.1 from 2.2.2.1 (2.2.2.1)
      Origin incomplete, metric 0, localpref 100, valid, external, best
! network 0.0.0.0
R2# show ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 12
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
  Not advertised to any peer
  1
    2.2.2.1 from 2.2.2.1 (2.2.2.1)
     Origin IGP, metric 0, localpref 100, valid, external, best

The difference here is that advertising a local network marks the origin as IGP, but a redistribution marks it as incomplete. Look at step 5 of the BGP best path selection algorithm to see that IGP is preferred over incomplete (even though by the time origin is compared, weight, local preference, and AS_PATH length have already been compared)

So, the moral of the story is to watch how your default routes go into your routing protocol, because depending on the protocol, it's handled differently.

Content Copyright Sean Walberg

Default routes

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I’m giving 2 talks on using Wireshark to expose VoIP problems at Sharkfest ‘08 (schedule). Details are sketchy, I think one of the talks is more of a hands on lab, the other is me talking. I’ve expanded on my techniques from the Linux Journal article I wrote on the topic.

Some other fascinating topics going on at the same conference, especially wireless analysis and performance monitoring. Hope to see you there.

Content Copyright Sean Walberg

I’ll be speaking at Sharkfest ‘08

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_25_see/command/reference/cli3.html#wp2168243

Today I was troubleshooting a problem at a newly renovated remote office with an IP phone that would power up but not boot. After swapping cables and phones, I remembered the TDR and tried it out:

Switch# test cable-diagnostics tdr interface g1/0/14
TDR test started on interface Gi1/0/14
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.
Switch#show cable-diagnostics tdr  interface gigabitEthernet 1/0/14
TDR test last run on: March 04 22:31:09

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/14  auto  Pair A     19   +/- 4  meters N/A         Open
                Pair B     4    +/- 4  meters N/A         Open
                Pair C     20   +/- 4  meters N/A         Open
                Pair D     20   +/- 4  meters N/A         Open

Looks like a problem on Pair B! You should have heard the suprise from the (telecom) guy on the other end of the line when I finally said "looks like a problem in the cabling, get the contractor to check pair B.

Content Copyright Sean Walberg

Using a 3750 to test cables

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.