Happy Birthday SR629007199!

It is with disappointment and frustration that I'm celebrating the 1st birthday of an unresolved Cisco support case. I'm not happy about it, plan to do some complaining in this post.

Now, don't get me wrong, I think the people at Cisco TAC are great. They're an absolutely first class support organization, the standard by which other vendors are judged, and they consistently give me great service.

In spite of their efforts, sometimes things just don't work out. This is one of those times.

I opened SR629007199 on January 31st 2014 after noticing a peculiar problem with an ISR G2 router: Servers couldn't receive packets intended for them because the router was screwing up their traffic. The Ethernet frames carrying these packets included the wrong destination MAC address, so the servers ignored them.

Specifically, the router was screwing up the IP->L2 address mapping required for IPv4 multicast packets. Instead of using 23 bits of the multicast group in the L2 header, that portion of the L2 header was all zeros. It looked like this:

Bogus dMAC on most of these frames

Those two HSRP packets originated by the router looked okay, and traffic from local sources was okay, but every frame containing packets which had transited the router was wrong. Instead of 01:00:5e:00:00:00, those frames should have been sent to 01:00:5e:78:01:24.

Now, to be fair, my configuration was a little "off in the weeds". Most folks don't route multicast in production, and I was using an unusual combination of interface types. I'm not surprised this is issue isn't widely complained about, even though it's 100% reproducible.

Still, it's a supported configuration, and a whole year later there's no indication that anybody at Cisco has even begun working on the problem. That's the reason for this blog post. I'm complaining about it publicly because I'm frustrated, and because the correct channel has gotten me nowhere for the last year.

TAC has explained to me repeatedly that their hands are tied. It seems that ISR product management doesn't care about this bug. Their "workaround" is suggesting that I disable CEF. Doing so resolves the framing issue, but opens another can of worms in the performance department. It's true, I can disable CEF only for multicast traffic, but that's basically all this router carries. It exists only to support a multicast-based voice/video collaboration application. I can't afford the performance penalty associated with process switching this traffic. TAC, to their credit, didn't take the workaround seriously either.

So, why do I think product management doesn't care? Well, the relevant bug is rated as Severity 6:

Severity: 6 Enhancement

According to Cisco, this severity rating indicates a request for new functionality or feature improvements:

Cisco Table of Severity Levels

I happen to think that framing IP packets into correctly formatted Ethernet frames is core functionality of an IP router, and not a request for new functionality. Looking over this table, I'd land this bug in severity 3 due to my "unusual circumstances."

TAC and I have been trying to make the case for this bug for a year now, but product management has totally ignored it.

So, I give up. Next time TAC calls to ask permission to close the support case, I'm going to let them.