OSPF: ABR as a Black Hole

Area Border Routers in an OSPF topology sometimes make promises they can't keep.  These promises come in the form of summary route advertisements (an advertised route is a promise to deliver packets, after all), and they're usually not a problem.

But sometimes they are a problem, and in some topologies they can result in packet loss that's sustained for several seconds.  Most discussions of packet loss and reconvergence in modern networks concern themselves with detection and correction of node or link failures.  Surprisingly, node recovery can be worse than the failure!  Here's the scenario:

R2 and R3 announce a summary default route into Area 1, rather than announcing R1's external prefixes.  They summarize because area 1 is a stub area, and that's just how it works.  ABRs for a stub area unconditionally inject a default route into the area.

If one of the ABRs fails, that's no problem.  R4 will detect the failure quickly (for some values of quickly), and redirect all traffic to R3.

But what happens when the ABR comes back online?  It will introduce a default route into area 1 unconditionally.  Even if it hasn't fully converged on the area 0 side.  Not good.

I labbed this up to see what would happen.  R1 is introducing 500 external prefixes (500 loopback interfaces with 'redistribute connected'), and R4 is pinging one of those prefixes.  I chose a prefix that CEF will hash to the path via R2 when both paths are available.

R4#debug ip routing

IP routing debugging is on

R4#ping

Protocol [ip]:

Target IP address: 10.255.254.20

Repeat count [5]: 100000

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 100000, 100-byte ICMP Echos to 10.255.254.20, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!

R2 was reloaded and a single ping was lost as R4 reconverged.  Not bad.  The outage duration was somewhere less than the 2 second ping timeout.  Debug output on the following lines shows the interface failure, OSPF neighbor loss and removal of the bad route.

*Mar  1 08:20:15.403: RT: del 0.0.0.0 via 10.0.24.2, ospf metric [110/2]

*Mar  1 08:20:15.403: RT: NET-RED 0.0.0.0/0

*Mar  1 08:20:15.539: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.24.2 on Serial3/3 from FULL to DOWN, Neighbor Down: Dead timer expired

*Mar  1 08:20:15.571: %LINK-3-UPDOWN: Interface Serial3/3, changed state to down

*Mar  1 08:20:15.571: RT: is_up: Serial3/3 0 state: 0 sub state: 1 line: 0 has_route: True

*Mar  1 08:20:15.571: RT: interface Serial3/3 removed from routing table

*Mar  1 08:20:15.571: RT: del 10.0.24.0/24 via 0.0.0.0, connected metric [0/0]

*Mar  1 08:20:15.571: RT: delete subnet route to 10.0.24.0/24

*Mar  1 08:20:15.575: RT: NET-RED 10.0.24.0/24

*Mar  1 08:20:16.571: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial3/3, changed state to down

*Mar  1 08:20:16.571: RT: is_up: Serial3/3 0 state: 0 sub state: 1 line: 0 has_route: False

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Now R2 is coming back up.  R4's debugs show the interface come online and the addition of the connected route:

*Mar  1 08:22:43.463: %LINK-3-UPDOWN: Interface Serial3/3, changed state to up

*Mar  1 08:22:43.463: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: False

*Mar  1 08:22:43.463: RT: SET_LAST_RDB for 10.0.24.0/24 NEW rdb: is directly connected

*Mar  1 08:22:43.467: RT: add 10.0.24.0/24 via 0.0.0.0, connected metric [0/0]

*Mar  1 08:22:43.467: RT: NET-RED 10.0.24.0/24

*Mar  1 08:22:43.467: RT: interface Serial3/3 added to routing table

*Mar  1 08:22:43.467: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: True

*Mar  1 08:22:44.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial3/3, changed state to up

*Mar  1 08:22:44.463: RT: is_up: Serial3/3 1 state: 4 sub state: 1 line: 0 has_route: True

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!

And the following debugs show R4's OSPF neighbor come back up, and the installation of the default route via R2.

*Mar  1 08:22:53.395: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.24.2 on Serial3/3 from LOADING to FULL, Loading Done

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

*Mar  1 08:22:56.823: RT: NET-RED 0.0.0.0/0

!!!!!!!!!!!!!

*Mar  1 08:22:58.967: RT: add 0.0.0.0/0 via 10.0.24.2, ospf metric [110/2]

*Mar  1 08:22:58.967: RT: NET-RED 0.0.0.0/0

*Mar  1 08:22:58.971: RT: NET-RED 0.0.0.0/0

Immediately after the addition of the default route at 08:22:58.967, pings began failing.  The U indicates that R2 dropped our ping, rather than delivering it to R1 and then sent us an ICMP unreachable message to let us know what happened.  R2 has promised to deliver packets to anywhere, but has not yet learned the route to 10.255.254.20 (a loopback on R1).  10 pings time out over about 20 seconds before connectivity is restored:

U.U.U.U.U.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.

Success rate is 99 percent (3204/3216), round-trip min/avg/max = 52/58/108 ms

The final lost ping happened because I killed off the ping process mid-ping.

Nifty, huh?  Reconvergence around a failed router happens in under two seconds, but reconvergence through a recovered router takes 20 seconds.

I'm not aware of a way to configure around this behavior in IOS.  Conditional default advertisement, or a stub gateway delay option would be nice, but I don't know how to configure such a thing in IOS.  Do you know how to prevent this problem?  Please let me know!