Assigning DMVPN tunnel interface addresses with DHCP

I posted previously about some of the inner workings of DHCP. The three key points from that post are critical building blocks for this discussion:

• DHCP requests get modified in flight by the DHCP relay.
• DHCP relay determines L2 destination by inspecting contents of relayed packets.
• DHCP clients, relays and (sometimes) servers use raw sockets because the end-to-end protocol stack isn't yet available.

The basic steps to converting a DMVPN from static address assignment scheme to dynamic are:

1. Configure a DHCP server. I'm using an external server¹ in this example so that we can inspect the relayed packets while they're on the wire.
2. Configure the hub router. There are some non-intuitive details we'll go over.
3. Configure the spoke router. Ditto on the non-intuitive bits.

My DHCP server is running on an IOS router (because it's convenient - it could be anywhere) and it has the following configuration:

    1     no ip dhcp conflict logging  
    2     ip dhcp excluded-address 172.16.1.1  
    3     !  
    4     ip dhcp pool DMVPN_POOL  
    5      network 172.16.1.0 255.255.255.0  

So, that's pretty straightforward.

The Hub Router has the following relevant configuration:

    1     ip dhcp support tunnel unicast  
    2     interface Tunnel0  
    3      ip dhcp relay information option-insert   
    4      ip address 172.16.1.1 255.255.255.0  
    5      ip helper-address 172.16.2.2  
    6      no ip redirects  
    7      ip mtu 1400  
    8      ip nhrp authentication blah  
    9      ip nhrp network-id 1  
   10      ip tcp adjust-mss 1360  
   11      tunnel source GigabitEthernet0/0  
   12      tunnel mode gre multipoint  
   13      tunnel vrf INTERNET  
   14      tunnel protection ipsec profile DMVPN_IPSEC_PROFILE  

Only lines 1, 3 and 5 were added when I converted the environment from static spoke tunnel addresses to dynamic addresses.

Line 1: According to the documentation, the ip dhcp support tunnel unicast directive "Configures a spoke-to-hub tunnel to unicast DHCP replies over the DMVPN network." Okay, so the replies are sent directly to the spoke. If you read my last post, you're probably wondering how this works. I promise, it's interesting.

Line 2: Configuring ip dhcp relay information option-insert causes the relay agent to insert the DHCP relay agent information option (option 82) into the client's DHCP packets. We'll look at the contents in a bit.

Line 3: I specified the address of the DHCP server with ip helper-address 172.16.2.2. Nothing unusual about that.

Okay, with those configuration directives, the hub is going to unicast DHCP messages to the client (good, because DMVPN provides a non-broadcast medium). How will the relay agent on the hub know where to send the server's DHCP replies? Last week's example was an Ethernet medium. The Ethernet MAC address appears in the OFFER, so the relay cracked it open in order to know where the OFFER needed to go. This OFFER will also have a MAC address, but it doesn't help. Rather than an un-solved ARP problem, this relay has an un-solved NHRP problem. It needs to know the NBMA (Internet) address of the spoke in order to deliver the OFFER.

So what did the relay stick in option 82 of the client's DISCOVER message, anyway? This:

    1       Option: (82) Agent Information Option  
    2         Length: 13  
    3         Option 82 Suboption: (9) Vendor-Specific Information  
    4           Length: 11  
    5           Enterprise: ciscoSystems (9)  
    6             Data Length: 6  
    7           Value: 11046d7786c2  

So, we've got option 82 with sub-option 9 containing a Cisco-proprietary 6-byte payload. I don't know what 0x11 and 0x04 represent, but I'm guessing it's "NBMA" and "IPv4"², because the next 4 bytes (0x6d7786c2) spell out the spoke's NBMA address (109.119.134.194).

The DMVPN hub / DHCP relay agent jammed the spoke's NBMA address into the DHCP DISCOVER that it relayed on to the DHCP server! The debugs spell it out for us:

    1     DHCPD: adding relay information option.  
    2     DHCPD: Client's NBMA address 109.119.134.194 added tooption 82  

 Now look at what the hub/relay did when the DHCP OFFER came back from the server:

    1     DHCPD: forwarding BOOTREPLY to client 88f0.31f4.8a3a.  
    2     DHCPD: Client's NBMA address is 109.119.134.194  
    3     NHRP: Trying to add temporary cache from external source with (overlay address: 172.16.1.3, NBMA: 109.119.134.194) on interface: (Tunnel0).  
    4     DHCPD: creating NHRP entry (172.16.1.3, 109.119.134.194)  
    5     DHCPD: unicasting BOOTREPLY to client 88f0.31f4.8a3a (172.16.1.3).  
    6     DHCPD: Removing NHRP entry for 172.16.1.3  
    7     NHRP: Trying to delete temporary cache created from external source with nex-hop UNKNOWN on interface: (Tunnel0).  

Just like the Ethernet case, the relay read the client's lower layer address info from the OFFER. Unlike the Ethernet case, the DMVPN relay gleaned this critical information from a field which had been inserted by the relay itself.

Somewhat different from the Ethernet case (which, according to debugs, did not populate the relay's ARP table with info from the OFFER), the DMVPN relay manipulated the NHRP table directly. Pretty cool.

A point about dual-hub environments: All of these DHCP gyrations are only possible because the DMVPN hub and spoke have already keyed the IPSec tunnel, and have an active Security Association with one another. In a dual hub environment, only one of the hubs will be able to talk to the client at this stage. Remember that the DHCP server unicasts the OFFER to the relay agent using the agent's client-facing address (172.16.1.1 in this case). If we have a second hub at, say, 172.16.1.2, it's likely that both hubs will be advertising the tunnel prefix (172.16.1.0/24) into the IGP. Routing tables in nearby routers won't draw a distinction between hub 1 (172.16.1.1) and hub 2 (172.16.1.2) when delivering the OFFER (unicast to 172.16.1.1) from the DHCP server, because both hub routers look like equally good ways to reach the entire 24-bit prefix. It is critical that the IGP has 32-bit routes so that traffic for each hub router's tunnel interface gets delivered directly to the correct box.

The Spoke Router sports the following configuration:

    1     interface Tunnel0  
    2      ip dhcp client broadcast-flag clear  
    3      ip address dhcp  
    4      no ip redirects  
    5      ip mtu 1400  
    6      ip nat enable  
    7      ip nhrp authentication blah  
    8      ip nhrp network-id 1  
    9      ip nhrp nhs dynamic nbma 109.119.134.213 multicast  
   10      ip tcp adjust-mss 1360  
   11      tunnel source FastEthernet4  
   12      tunnel mode gre multipoint  
   13      tunnel vrf INTERNET  
   14      tunnel protection ipsec profile DMVPN_IPSEC_PROFILE  
   15     ip route 172.16.1.0 255.255.255.0 Tunnel0  
   16     ip route 0.0.0.0 0.0.0.0 FastEthernet4 dhcp  

Line 2: This option clears the broadcast flag bit in the DISCOVER's bootp header. This doesn't affect delivery of the DISCOVER, because of the multicast keyword on line 9. Clearing the broadcast flag in the DISCOVER also results in no broadcast bit in the OFFER message which needs to be relayed to the client. This is important because the hub doesn't have an outbound multicast capability. The DHCP OFFER will never get delivered in over the NBMA transport if the broadcast bit is set.

Line 3: Pretty self explanatory

Line 9: I'm partial to this method of configuring the NHS server (as opposed to two lines: one specifying the tunnel address of the NHS, and one specifying the mapping of NHS tunnel IP to NMBA IP. At any rate, there's a critical detail: the multicast keyword must be present in either configuration. Without it, the spoke won't have an NHRP mapping with which to send the DISCOVER messages upstream.

Line 15: This is a funny one, and might not be required in all environments.

First, consider line 16. There is a default route in the global table via the DHCP-learned next hop in vrf INTERNET. This is here to facilitate split tunneling: Devices behind this DMVPN spoke can access the Internet via overload NAT on interface Fa4.

Next, remember what the OFFER message looks like: It's an IPv4 unicast destined for an address we don't yet own.

Look what comes out FastEthernet 4 if we don't have the route on line 15 in place:

 poetaster:~ chris$ tcpdump -r /tmp/routed.pcapng   
 reading from file /tmp/routed.pcapng, link-type EN10MB (Ethernet)  
 16:25:09.481759 IP 172.16.1.1.bootps > 172.16.1.3.bootpc: BOOTP/DHCP, Reply, length 300  
 16:25:12.677629 IP 172.16.1.1.bootps > 172.16.1.3.bootpc: BOOTP/DHCP, Reply, length 300  
 poetaster:~ chris$  

Those are our DHCP OFFERS, and we're spitting them out toward the Internet!

What's going on here? Well, we've got an IPSec SA up and running with our DMVPN hub. We don't yet have an IP address, but IPSec doesn't care about that. It's not a VPN, it's more of a transport security mechanism, right? An encrypted packet rolls in from the hub, matches the SA, gets decrypted and then... Routed! The DHCP client process never saw it. The OFFER missed the raw socket (or whatever Cisco is doing) mechanism entirely!

Line 16 pushes the wayward OFFER back toward the tunnel interface, where the DHCP client implementation will find it. This route doesn't need to match the tunnel interface exactly, it just needs to be the best match for the address we will be offered. I used a 24-bit route to match the tunnel, but these would have worked too:

• 172.16.1.3/32 (requires me to know my address in advance)
• 172.16.0.0/12 (requires that I don't learn a better route to my new IP via some path other than Tu0)

Maybe there's a way to handle the incoming DHCP traffic with PBR? That would be an improvement because this static route detail is the only place in the configuration which requires the spoke to know anything about the prefix running on the DMVPN transport he'll be using.

---

1 Cisco's documentation indicates that the DHCP server cannot run on the DMVPN hub. Take that indication with some salt because (a) That is an IOS XE command guide, and IOS XE doesn't have some of the indicated commands. (b) Running the DHCP server locally seems to work just fine.^↩

2 Maybe "IPv4 NBMA" and "length" ? Eh. That's why it's called a vendor proprietary option. ^↩