ICMP Redirects Broke the Voice Network – Understanding Traffic Flows

Some time ago Phil Gervasi tweeted about understanding traffic flows and it got me thinking about situations where I did not quite understand how the production network was functioning. Phil’s tweet was more about fault tolerance, but I think it applies to general Network Engineering. If you do not understand how data flows, it becomes very difficult to troubleshoot your network.

I remembered a time where I was completely perplexed on a traffic flow situation. I had completed a L3 upgrade, and a couple days later I was told that call quality had become poor at a remote site. Initially I requested the team check with their WAN provider, as they could not pinpoint a start time and the remote site was supposed to be segmented off and not utilizing the recently upgraded equipment. Not to mention the WAN connection had been notoriously unreliable.

A few days went by and I was contacted by the site team again. They had indicated to me that the WAN provider tested the circuit, and the team also had the remote site router replaced but call quality was still poor. I decided to ask for router access so I could poke around. What I found was that consultants will sometimes take the easiest path, not the correct path.

After digging around in the router config and drawing a quick diagram I quickly came to the realization that a consultant had stuck a single static route in the voice router – a route that was pointed back to the device I had just upgraded. Even though the device and destination were on the same subnet, the static route was sending everything to my router.

Thanks to a new, default CoPP config, my router began dropping traffic because the ICMP redirects were being ignored. Essentially my router thought it was being attacked, so it was dropping traffic causing poor voice quality. As a workaround, I turned off ICMP and IP redirect. Soon after call quality was back to normal.

I learned two things that day – static routes are the devil and you have to understand how traffic flows on your network.

If you are interested in seeing how CoPP is affecting your network run:

 sh ip policy-map

Additionally, I would challenge you to try Phil’s follow-up tweet as well –

I promise you will learn something new about your network and become a better Engineer. I know I became a better Engineer that day.

 

Leave a Reply

Your email address will not be published. Required fields are marked *