Posted on Sunday, 16th April 2006 by sean
Multilink PPP is an add on to PPP encapsulation that lets you bond multiple links into one. It can be used over asynchronous dialup to bond multiple modem or ISDN connections, or to bond multiple synchronous serial lines such as T1s.
Read the rest of this entry…
Posted in General | Comments (0)
Posted on Thursday, 6th April 2006 by sean
Due to my own laziness, I let my CCNP expire over a year ago, which means I have to start at the bottom. I finally passed the CCNA today.
Read the rest of this entry…
Posted in General | Comments (3)
Posted on Monday, 27th March 2006 by sean
Turns out we had a bad CSU/DSU on a T1 link recently, the thing that made it more difficult was that it was internal to the router (WIC-1DSU-T1)
Troubleshooting serial links is done by testing different parts of the links by looping endpoints.
The basic procedure:
http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml
The difference between loopback dte, loopback remote, and loopback line with pictures.
http://www.cisco.com/en/US/products/hw/routers/ps233/products_tech_note09186a0080093c56.shtml
How to reset the CSU:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1826/products_configuration_guide_chapter09186a00800d9b62.html#xtocid22
Posted in General | Comments (0)
Posted on Friday, 10th March 2006 by sean
I’m a self professed Cisco bigot. I use the term “bigot” because that’s what the Nortel sales guy called me and the name stuck. I’ve also heard the term “customer evangelist” thrown around by marketing people — a customer who is happy with the company, and tells others about it.
Why do I willingly pay twice the price (granted, it’s not my money) for the same hardware? The answer is support.
Read the rest of this entry…
Posted in General | Comments (3)
Posted on Thursday, 9th March 2006 by sean
It’s rare that I open a case with TAC, I’m usually able to figure things out with the help of the docs or by working at it. But this past week, I’ve been involved with several cases:
Upgrade problems – After swapping the flash in a 3660, we couldn’t store an image on the new flash via xmodem. The solution ended up being for us to copy the image to DRAM, boot to that, format the filesystem, then copy the image to flash
ATM problems – After the upgrade, ATM failed. Turns out there’s a bug in the version we were upgrading from that enabled ATM payload scrambling no matter what the config said. After the upgrade, it respected the config, turned scrambling off, which meant the router couldn’t talk to its peers. See: http://www.cisco.com/en/US/tech/tk39/tk49/technologies_tech_note09186a00800d73b7.shtml#knownissue_atm_scramble
CiscoWorks won’t sync archive – Even though my CW2K server could log in to all the devices, it couldn’t pull down the configs. I still don’t understand why, but after telling CW2K to use TFTP it worked. I still ran into a bug where many switches are shown as “partial failures” because of a bug in CW2K, even though it succeeded.
CiscoWorks Netconfig jobs don’t work on switches – Still going through this one with TAC. I can run a netconfig job on routers just fine, but not on switches
policy-map editing hangs the exec session – after the above upgrade, whenever I try to create or edit a policy-map, the exec session hangs. I just opened the case on this one.
I also noticed that the CPU on the router climbed about 10%, might need a case on that to help investigate.
Posted in General | Comments (0)
Posted on Thursday, 23rd February 2006 by sean
I got a call from someone in the company yesterday whose husband is getting retrained and is considering a career in network administration. They asked if I could meet with him to tell him about the career. I’m happy to, but what do I say?
Network admin is such an inaccurate term. In both the places I’ve worked so far the network guys end up having their hands in everything because they’re the ones with the most aptitude for figuring things out. I mean three weeks ago our Notes server died and with four server guys watching us, the network guys rebuilt it. Just because it ran Linux and it was outside their comfort zone.
People often joke that my kids were probably born with a router in their hands, but I often think they should do something else. This is probably a standard parental desire to want better for one’s offspring, but would I wish this career on another? After all, I can’t imagine doing anything else.
Posted in General | Comments (2)
Posted on Thursday, 26th January 2006 by sean
Just testing posts from the blackberry. I seem to get good ideas on the way to and from work, maybe posting them immediately will help!
Posted in General | Comments (0)
Posted on Sunday, 22nd January 2006 by sean
I just moved from Movable Type to Wordpress for this blog, mostly to get a handle on the comment spam that has been plaguing me. I made the switch sooner than I’d like, so please let me know if you run into any problems viewing the site, or finding old content.
Sean
Posted in Uncategorized | Comments (2)
Posted on Monday, 16th January 2006 by sean
An interesting thing happened last week that illustrates how understanding the behaviour of a packet travelling through a network explains the cause of some rather odd traffic patterns.
It was noticed that all active ports on a certain switch would simultaneously see traffic spikes in excess of 8Mbit/s, even though these ports were normally dormant. I don’t have the pictures anymore, but it’s quite obvious that something was up. What follows is a slightly modified version of what happened, edited to make it more clear about what’s really going on.
Luckily the event happened again while we were able to watch it. Using Ethereal, we saw one side of a conversation being flooded out all the switch ports.
The behaviour of a switch is to capture the source MAC of a frame and associate it with the port it came in on within the Content Addressable Memory (CAM) table. Whenever a frame is to be sent, the CAM table is consulted for the destination MAC, and if found, the frame is only sent out that port. If the MAC isn’t found, the frame is sent out all active ports in the VLAN, except the frame from which it was received.
True to spec, the MAC of the local host could not be found with the show mac-address-table command. (hint, show mac-address-table | include XXXX, where XXXX is the last 4 nybbles of the MAC, is a fast way to search)
In this situation, LocalHost was talking to RemoteHost over the WAN. The port flooding was being seen on the switch connected to the WAN router, while LocalHost was on another switch, but on the same VLAN as the WAN router.
From layer 3, everything seemed to be with the RemoteHost->LocalHost communication, but at layer 2 it was being flooded.
At this point a diagram would help. Please pardon the ASCII art until I can get a Visio done up! Here is a cut down version of the network:
RemoteHost --- WAN RTR ----[sw1]-----[sw2]-----Core RTR
|
LocalHost
It was sw1 where the port flooding was occurring.
Turning our attention to layer 2, we looked at the MAC address of the packets. LocalHost was sending its packet to Core RTR, while the packet hitting the WAN came from Core RTR. Looking closely at the configuratin of Core RTR, no ip redirects was configured on the VLAN. Turning this on fixed the problem.
So what was happening?
LocalHost’s gateway was Core RTR. To send a packet, LocalHost consults its routing table, determines the destination is reachable by the gateway, and sends out the packet as follows:
L2: LocalHost->Core RTR
L3: LocalHost->Remote Host
Keeping track of the CAM tables will be helpful:
SW1: empty
SW2: LocalHost
The Core RTR then picks up the packet, looks at the destination IP address, and sends it along.
L2: Core RTR->WAN RTR
L3: LocalHost->Remote Host
SW1: Core RTR
SW2: Core RTR, LocalHost
The normal behaviour in this case would have been to send the packet, but also issue an ICMP Redirect to LocalHost, telling it that he’s better off talking to the WAN RTR for that particular destination. This is repressed with no ip redirects
The WAN router gets the packet and sends it out the WAN and to the destination. The response comes back to the WAN router. However, since the WAN router is on the same subnet as LocalHost, it can send the packet directly:
L2: WAN RTR->LocalHost
L3: RemoteHost->LocalHost
Looking at the CAM tables in the previous step, LocalHost isn’t known to SW1 so it floods the packet to all the switchports. As long as this asymetric path exists, the flooding would happen. Even if LocalHost were to do something as simple as ping WAN RTR, SW1 would learn about LocalHost’s MAC and temporarily
(5 minutes by default) eliminate the port flooding.
In conclusion, solving this problem required understanding the behaviour of a router beyond simply “routing packets between interfaces”. Also, it required viewing layer 2 information in the sniffer, something that most don’t show by default (-e adds this information in tcpdump)
Posted in General | Comments (0)
Posted on Monday, 26th December 2005 by sean
Routing protocols, especially EIGRP, are very good at finding ways around network faults. I remember one time where we had a LAN failure, but a router on each side of the break happened to be connecting to the same site over two different WAN links, and EIGRP shunted all the LAN traffic through that office (via a low speed link) until we fixed the break. Though the LAN was technically up, the bandwidth available was so low that it didn’t work well.
This comes at the expense a larger query scope for the DUAL algorithm, which results in more Stuck in Active (SIA) routes. In many cases you know if a particular router will never transit traffic (ie all traffic comes from, or to, networks attached to the router). These are called stub zones (or areas). OSPF has these concepts, and so does EIGRP, it’s just that it’s not widely known.
R0 and R1 are distribution routers connected to the corporate backbone. R2 is connected over the WAN, and is a remote office.
From R2’s perspective, any routes coming from R0 should not be advertised back to R1, and vice-versa. Likewise, when going active on a router, R0 and R1 shouldn’t ask R2. But, by default, all of this happens to poor R2.
By configuring R2 as a stub you can choose what type of routes to advertise: connected, static, summary, or a combination of the three. You can also choose to advertise no routes, and rely on the distribution layer to handle this for you.
r2(config)#router eigrp 1 r2(config-router)#eigrp stub ? connected Do advertise connected routes receive-only Set IP-EIGRP as receive only neighbor static Do advertise static routes summary Do advertise summary routes r2(config-router)#eigrp stub
On R0, we can now see that the neighbour is known as a stub, and that since I didn’t specify what type of routes to advertise, the defaults are connected and summary routes (ie not redistributed static routes). R0 also knows not to make any queries to R2 in the event that it goes active on a route.
r0#show ip eigrp neighbors detail
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 10.2.2.1 Se0.131 14 00:00:51 444 2664 0 39
Version 12.1/1.2, Retrans: 0, Retries: 0
Stub Peer Advertising ( CONNECTED SUMMARY ) Routes
Suppressing queries
1 10.50.0.2 Et0 10 00:01:22 23 200 0 40
Version 12.2/1.2, Retrans: 3, Retries: 0
Cisco has a document about EIGRP stub routing, and Optimal Routing Design has a section on it, which is where I originally learned about this feature.
Posted in Routing | Comments (0)
I'm a happy Linode customer. This is a Linode 540 VPS.