Reverse Telnet by IP address

There's lots of pages out there detailing how to use your 2511 router or NM-16A/NM-32A for reverse telnet (ugh, I don't care for that term) duty.

Generally they end with each serial interface being associated with a TCP port on the router.   The TCP port for each interface is the interface's "line number" + 2000.

So, if your NM-16A represents lines 33-48, you'd be able to access the console attached to the 3rd serial interface by with "telnet <router> 2035"

While this is adequate for most labs where things get moved around a lot, I find it's difficult to keep track of these port numbers in a long-term deployment.  Especially when there are multiple routers doing this job:  The overlap created by multiple devices means you can't just map the device names in your services (files/NIS/LDAP) database.

My solution is to map each interface to a separate IP address, then create DNS records for every one of them.  Actually, two DNS records for every interface:  A permanent "A" record that references the device and port (terminalserver4-port3.mydomain.com), and a "CNAME" record that references the managed device's console port (switch3-console).  The CNAME points at the A record.

Accessing each serial line by a unique IP address makes this possible, but it requires some address translation.  Here's the relevant configuration, assuming you've already got reverse telnet working:

Configure a loopback interface with a netmask roomy enough for all of your serial interface IPs:


interface Loopback0
 ip address 192.168.4.1 255.255.255.0
 ip nat inside
 ip ospf network point-to-point

This interface is "inside" from a NAT perspective.  The point-to-point line is required because without it, OSPF always advertises loopback interfaces with a 32-bit mask.  Skin that cat however you see fit.

Next, set your Ethernet interface for NAT outside:

interface Ethernet0

 ip nat outside

Now, setup the NAT rules to forward incoming connections.  The connections arrive on unique IPs, all port 23 (no ssh support on my router), and are forwarded to the loopback interface on the unique per-line TCP port:

ip nat inside source static tcp 192.168.4.1 2033 192.168.4.11 23

ip nat inside source static tcp 192.168.4.1 2034 192.168.4.12 23

ip nat inside source static tcp 192.168.4.1 2035 192.168.4.13 23

ip nat inside source static tcp 192.168.4.1 2036 192.168.4.14 23

ip nat inside source static tcp 192.168.4.1 2037 192.168.4.15 23

ip nat inside source static tcp 192.168.4.1 2038 192.168.4.16 23

ip nat inside source static tcp 192.168.4.1 2039 192.168.4.17 23

ip nat inside source static tcp 192.168.4.1 2040 192.168.4.18 23

ip nat inside source static tcp 192.168.4.1 2041 192.168.4.19 23

ip nat inside source static tcp 192.168.4.1 2042 192.168.4.20 23

ip nat inside source static tcp 192.168.4.1 2043 192.168.4.21 23

ip nat inside source static tcp 192.168.4.1 2044 192.168.4.22 23

ip nat inside source static tcp 192.168.4.1 2045 192.168.4.23 23

ip nat inside source static tcp 192.168.4.1 2046 192.168.4.24 23

ip nat inside source static tcp 192.168.4.1 2047 192.168.4.25 23

ip nat inside source static tcp 192.168.4.1 2048 192.168.4.26 23

To connect to the 5th serial interface, I'd telnet to 192.168.4.15.

TFTP Oddities

Like RS-232, TFTP is less and less used these days, but remains an absolutely critical protocol for network engineers.

Here I'll present a couple of TFTP implementation oddities that sometimes break TFTP.

TFTP Uses Ephemeral Ports

The bulk of the data flow between a TFTP client and server uses ephemeral ports on both ends.  This is strange behavior.  Most applications use ephemeral ports on the client side, but the server responds from its well known port number.  Here's an example of a normal application (a DNS query):

13:12:13.497988 IP 192.168.1.91.56119 > 192.168.1.1.53:  54600+ A? www.google.com. (32)

13:12:13.583574 IP 192.168.1.1.53 > 192.168.1.91.56119:  54600 5/4/4 CNAME www.l.google.com.,[|domain]

The client originated the query from an ephemeral port (56119), and sent it to the server's well known port (53).  The server's response was symmetric: 53 -> 56119.

TFTP servers don't generally behave this way.  Consider this TFTP transaction:

12:55:33.188470 IP 192.168.1.9.34617 > 192.168.1.91.69:  22 RRQ "myfile.dat" netascii
12:55:33.189962 IP 192.168.1.91.42829 > 192.168.1.9.34617: UDP, length 516
12:55:33.190060 IP 192.168.1.9.34617 > 192.168.1.91.42829: UDP, length 4
12:55:33.190088 IP 192.168.1.91.42829 > 192.168.1.9.34617: UDP, length 92
12:55:33.190190 IP 192.168.1.9.34617 > 192.168.1.91.42829: UDP, length 4

The client originated the transaction from an ephemeral port (34617), and sent it to the server's well known port (69).  But the server's response was NOT symmetric.  Rather than replying from port 69, the server spawned a new ephemeral port (42829) when replying to the client.  After the initial request packet, the four packets responsible for the file transfer (data, ack, data, ack) were exchanged between two ephemeral ports.

This is strange behavior, and most flow-mangling smart devices (firewalls, NAT engines, reflexive ACLs, etc...) won't recognize the second or later packets as being related to the first.  Unlike active mode FTP, TFTP doesn't include an application header describing what's going to happen next.  Accordingly, there's nothing for a protocol-aware firewall module to monitor in order to open a pinhole for return traffic.  TFTP is a perfect candidate for a man-in-the-middle attack.

I'm not aware of any requirement for TFTP servers to act this way, but most do.  My guess about the reason for the behavior is this:  As shown here, the TFTP server merely needs to fork() itself for each client request.  The new thread opens its own socket to handle the client request, then exits when done.  Without the separate thread and socket, a single server thread would have to keep track of all clients simultaneously, and would have to demultiplex incoming client packets for each flow (because they'd all flow in the single connectionless UDP socket).

TFTP Supports Broadcast Servers

Sort of.
I expect lots of people to run into this issue as enterprises upgrade their Cisco wireless environments to 802.11N, the old gear floods onto eBay, and people attempt to load the autonomous AP code.  There's a peculiarity with the TFTP implementation on the Cisco Aironet autonomous-capable access points.  Consider my AIR-AP1231G-A-K9 trying to download its IOS software from the TFTP server.  The AP's console reports:

process_config_recovery: set IP address and config to default 10.0.0.1

process_config_recovery: image recovery

image_recovery: Download default IOS tar image tftp://255.255.255.255/c1200-k9w7-tar.default

examining image...

extracting info (274 bytes)

Premature end of tar file

ERROR: Image is not a valid IOS image archive.Loading "flash:/c1200-blah-blah-blah

What's going on here?  I'm not the first to notice this problem, but I haven't seen an explanation anywhere yet.  First, notice that the client is hitting its TFTP server at 255.255.255.255.  Cisco's documentation of this process directs that the TFTP server be configured with an address between 10.0.0.2 and 10.0.0.30.  That's because the AP will be running with 10.0.0.1, and it doesn't know exactly where server will be, so it hits the all-ones address.  Why we're not supposed to use 10.0.0.31, I don't know.  On the wire we get this:

12:50:54.498194 IP 10.0.0.1.1024 > 255.255.255.255.tftp:  31 RRQ "c1200-k9w7-tar.default" octet

Next, the server unicasts the first chunk of the IOS image to the AP (ephemeral port to ephemeral port again):

12:50:54.547794 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516

So far, so good, but we're getting to the point where things come off the rails.  The AP, having just received a packet from 10.0.0.5, should know the server's address.  But it sends the TFTP acknowledgement for the first chunk of the file to the broadcast address, not directly to the server:

12:50:54.606812 IP 10.0.0.1.1024 > 255.255.255.255.56216: UDP, length 4

The TFTP server doesn't hear these ACKs, and both ends endlessly retransmit their last packet.  I'm not clear on whether this should be reasonably expected to work, but...

• It flies in the face of the thread-per-client model which I believe is behind the port symmetry issue
• My antique Xyplex terminal servers use broadcast mode TFTP, and get the replies right
• It doesn't work

Here's the whole exchange, with retransmissions tagged in red:

12:50:54.498194 IP 10.0.0.1.1024 > 255.255.255.255.tftp:  31 RRQ "c1200-k9w7-tar.default" octet
12:50:54.547794 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516
12:50:54.606812 IP 10.0.0.1.1024 > 255.255.255.255.56216: UDP, length 4
12:50:55.543296 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516
12:50:57.543421 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516
12:51:01.543679 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516
12:51:01.544614 IP 10.0.0.1.1024 > 255.255.255.255.56216: UDP, length 4
12:51:06.626855 IP 10.0.0.1.1024 > 255.255.255.255.56216: UDP, length 4
12:51:09.544176 IP 10.0.0.5.56216 > 10.0.0.1.1024: UDP, length 516
12:51:14.627345 IP 10.0.0.1.1024 > 255.255.255.255.56216: UDP, length 4

In 20 seconds, the first block of the file got sent 5 times, and was (poorly) acknowledged 4 times, before both ends gave up.

That's what happends with tftp-hpa 0.49, anyway.  Philippe Jounin's TFTPD32 v3.33 doesn't seem to mind this sort of thing:

13:41:02.083839 IP 10.0.0.1.1024 > 255.255.255.255.tftp:  31 RRQ "c1200-k9w7-tar.default" octet
13:41:02.139137 IP 10.0.0.2.4068 > 10.0.0.1.1024: UDP, length 516
13:41:02.197885 IP 10.0.0.1.1024 > 255.255.255.255.4068: UDP, length 4
13:41:02.198033 IP 10.0.0.2.4068 > 10.0.0.1.1024: UDP, length 516
13:41:02.211902 IP 10.0.0.1.1024 > 255.255.255.255.4068: UDP, length 4
13:41:02.211987 IP 10.0.0.2.4068 > 10.0.0.1.1024: UDP, length 516
13:41:02.213473 IP 10.0.0.1.1024 > 255.255.255.255.4068: UDP, length 4

So, when installing new code on your Aironet APs, be careful which TFTP daemon you use.