ICMP Covert Channel for IOS

I wrote a quick-and-dirty covert channel via ICMP for IOS routers.

The channel in question isn't super covert. It's all in plaintext and is quite noisy because it only delivers a single byte of message payload per ping. But it gets messages from routers to the listener via pings, and that was the objective. I expect it to be useful when diagnosing IPSec issues behind unknown overload NATs.

It lives here.

Invoke it on a router like this:

Router#tclsh flash:sender.tcl <target> testing 1 2 3

It will then send 14 pings (13 for the characters in 'testing 1 2 3' plus an <EOM> terminator) to the target machine.

The listener functions as a packet sniffer, so it requires root access. It prints out a line per incoming message, preceded by the sender's IP address:

# /tmp/listener.py 

192.168.5.5 testing 1 2 3

Inspecting SCEP enrollment traffic

SCEP is a protocol which facilitates client enrollment with a Certificate Authorities (CA), delivery and renewal of certificates and delegation of identity verification from a CA to a trusted Registration Authoritie (RA)

A project I'm working on requires me to generate a Certificate Signing Request (CSR) on behalf client which doesn't exist yet, and deliver of those requests to the CA via an RA that I'm building. I'll then set aside the certificate and keys for installation onto the client system when it becomes available.

It seemed like ripping apart a request from a real client, as delivered by a real RA would be a good place to start, so that's what I did. I set up a CA (R1), an RA (R2) and a client (R3), performed the enrollment and captured the traffic between the R2 and R1.

There's a nice diagram detailing how a client delivers its  to a CA on this Cisco page, so have a quick peek at the breakdown listed under Client Enrollment there.

A CSR delivered by an RA (rather than the client) is similarly encapsulated, except that both of the PKCS7 functions are performed by the RA (with the RA's private key), rather than the client.

Cisco's diagram looks fairly straightforward, but it took me a while to work out the handful of openssl incantations that follow.

The first step was to collect the data delivered over HTTP using a sniffer. The RA's HTTP GET request looks like this:

 /cgi-bin/pkiclient.exe?operation=PKIOperation&message=MIIJUwYJKoZI...etc...  

The value of the message field is a few KB of URL encoded text, so the first step was to snag that text, stick it in a file I called RA->CA.raw

The text is URL encoded, so a quick substitution restored the newlines, '+', '/' and '=' characters to their usual form, leaving behind a nice base64 encoded file with 64 characters on each line. I saved it as RA->CA.b64

Decode that base64 data to its binary form:

 chris$ base64 --decode 'RA->CA.b64' > 'RA->CA.bin'  

The result is a PKCS7 signedData blob in DER format. You can browse the ASN.1 info for the request I'm working on here. This blob will have the certificate of the signer included.

In a client -> CA request, the client uses his own certificate to sign re-enrollment requests, and uses a self-signed certificate to bootstrap the initial enrollment. In this case (RA -> CA), the data is signed by the RA, so it's the RA's certificate that's included. You can see it at offset 1116 in the ASN.1 parser  linked in the previous paragraph.

Let's extract the RA's certificate:

 chris$ openssl pkcs7 -inform der -print_certs -in 'RA->CA.bin' | openssl x509 -out RA_cert.pem  

And let's have a look at it:

 chris$ openssl x509 -noout -in RA_cert.pem -issuer -subject  
 issuer= /CN=CA  
 subject= /OU=ioscs RA/unstructuredName=R2/serialNumber=4279256517  

The message was signed by the RA and we have that certificate. The RA's certificate was signed by the CA and we don't have that one. We're going to need the CA's certificate in order to extract the payload of this PKCS7 signedData bundle, because without it the trust chain is incomplete so the message won't validate. The CA in this case is a Cisco IOS router. Export the certificate in PEM format:

 R1(config)# crypto pki export ROOT-CA pem terminal  

Save the resulting text in CA_cert.pem and validate/extract the payload from the PKCS7 signedData blob sent by the RA to the CA:

 chris$ openssl cms -inform der -verify -CAfile CA_cert.pem -in 'RA->CA.bin' -out 'RA->CA.payload1'  
 Verification successful  

Great. Now we've got the payload from the PKCS7 signedData blob. What's in there? Another PKCS7, of course. This time it's envelopedData (encrypted). ASN.1 decoder.

This data is encrypted with the CA's public key, so only the CA (holder of the corresponding private key) can read it. We need the CA's private key. Export it from the CA:

 R1(config)# crypto key export rsa CA pem terminal 3des mypassphrase  

Save the resulting text (just the bits between the begin/end private key markers) in CA_key.3des. Strip the passphrase for sanity's sake. This is a test environment after all!

 chris$ openssl rsa -in CA_key.3des -out CA_key.pem -passin pass:mypassphrase  

Now we can use the CA's private key to decode the message sent to the CA:

 chris$ openssl cms -decrypt -in 'RA->CA.payload1' -inkey CA_key.pem -inform der -out 'RA->CA.payload2'  

So... What's in this RA->CA.payload2 file? Back to the ASN.1 decoder... Holy cow, it's a Certificate Signing Request!

 chris$ openssl req -in 'RA->CA.payload2' -inform der -noout -subject  
 subject=/unstructuredName=R3/serialNumber=4279256517  

Next steps for me are to run all of these steps backward so that I can deliver such a package to a CA for signing, but without having a real client or RA system. I'll start by generating keys, then a CSR, then encrypting the CSR, signing it with an RA's certificate, etc...

udevadm, systemd and a barcode scanner

I've been fooling around with a Symbol LS2208 barcode scanner attached to a CentOS 7 machine as part of a network automation project. I learned a bit about the scanner, udev and systemd along the way.

The LS2208

I chose the LS2208 because there were lots of them on eBay and because documentation was available. So far I'm happy with the LS2208, but wish it didn't require a physical PC to be nearby. A USB Anywhere box may be in my future (nope, Windows only). If I'd been able to find a WiFi scanner that would POST scans directly to a REST API over TLS, I'd have gone with that instead, but it seems that this guy and I are out of luck in that regard. I've got zero interest in fooling around with WinCE or similar mobile devices with built-in scanners.

The LS2208 gets configured by scanning barcodes. Special codes found in the manual. So far, the ones I've found most interesting are:

• Set Factory Defaults
• Simple COM Port Emulation
• Low Volume
• Beep on <BEL> (still need to fool with this - seems like it could provide useful feedback to the operator)
• Do Not Beep After Good Decode

By default the scanner appears with USB vendor/product codes 0x05e0/0x1200 which makes it emulate a USB keyboard. Fun to play with, but not how I want it to work for my project.

Scanning the Simple COM Port Emulation barcode found in the manual changes its USB personality to 0x05e0/0x0600, so that it shows up as an hidraw Linux device. I can't find any indication that it's a serial port, though the symbolserial device driver loads in CentOS 7.

With each scan, the device at /dev/hidrawX produces 64 bytes consisting of:

• A single byte indicating the length of the scanned data. UPC codes, for example, are 12 bytes long so they always produce 0x0C as the first byte.
• The scanned data
• Padding with <NUL> characters up to 64 bytes in total

We can take a quick look at the scan data with: od -t x1 < /dev/hidraw0.

A tiny bit of python

This little program reads from the scanner and spits out the result:

 #!/usr/bin/env python  
 import argparse  
 import sys  
   
 defaultDevice = '/dev/scanner'  
   
 parser = argparse.ArgumentParser()  
 parser.add_argument('-d', '--dev', dest='d', default=defaultDevice,  
           help='Scanner device; defaults to '+defaultDevice)  
 parser.add_argument('-o', '--out', dest='o', help='Output file')  
 args = parser.parse_args()  
   
 inDev = open(args.d, 'rb')  
 inDev.flush()  
   
 if args.o:  
  outFile = open(args.o, 'a')  
 else:  
  outFile = sys.stdout  
   
 while True:  
  inBytes = inDev.read(64)  
  length = ord(inBytes[:1])  
  value = inBytes[1:length+1]  
  outFile.write(str(length)+"\t"+value+"\n")  
  outFile.flush()  

udev

Not wanting to count on the scanner always landing on /dev/hidraw0, the next task was to write a udev rule to make it easy to find. I wound up with the following in /etc/udev/rules.d/10-scanner.rules:

 KERNEL=="hidraw[0-9]*", ATTRS{idVendor}=="05e0", ATTRS{idProduct}=="0600", GROUP="barcode", ACTION=="add", SYMLINK="scanner", TAG+="systemd", ENV{SYSTEMD_WANTS}="BarcodeScanner@%N.service"  

Essentially, I'm matching hidraw devices with the appropriate vendor and product codes, setting the resulting device file to be owned by group barcode (a group I created for use by the scanner project) and creating a symlink: /dev/scanner -> /dev/hidraw0

When the device appears, udev will ask systemd will kick off the BarcodeScanner unit with an argument (%N in the udev rule) indicating the path to the scanner device.

systemd
Finally, we need the systemd unit file to go with that udev rule. Here's my /etc/systemd/system/BarcodeScanner@.service unit file:

 [Unit]  
 Description=Simple Symbol Scanner  
 StopWhenUnneeded=yes  
   
 [Service]  
 Type=simple  
 ExecStart=/tmp/scan.py -d %I -o /tmp/scanned.txt  
   
 [Install]  
 WantedBy=multi-user.target  

The StopWhenUnneeded directive causes systemd to kill (SIGTERM, then SIGKILL if needed) the scan.py loop when nothing "wants" it. The %I evaluates to the device file passed in by udev.

Type=simple tells systemd that it shouldn't expect this unit to properly daemonize itself.

Poke systemd so that it notices the new unit file:

 sudo systemctl daemon-reload  

Now, when the scanner's USB cable is plugged in, the python code starts up and gets pointed (-d argument) at the scanner to start logging bar codes to /tmp/scanned.txt. When the scanner is unplugged, the BarcodeScanner unit is no longer "wanted" by anything, so systemd kills it off.

Cisco Debug Persists Through Reboot

Normal boot time messages from a C881 router look something like this:

 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode   
   
 Readonly ROMMON initialized  
   
   
 IOS Image Load Test   
 ___________________   
 Digitally Signed Production Software   
 Self decompressing the image : ###<snip>### [OK]  
   

But there's one router in the fleet which does this instead:

 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode  
   
 Readonly ROMMON initialized  
 Using monlib version 2  
 Using version info 2  
   
  dfs_openfile: Opening file.....  
  dfs_openfile: Opened file / with fib = 4019e5c  
 Reading cluster = 126, offset = 0, nsecs = 8  
 Reading cluster = 133, offset = 0, nsecs = 8  
 Reading cluster = 17013, offset = 0, nsecs = 8  
 Reading cluster = 17458, offset = 0, nsecs = 8  
 Reading cluster = 18056, offset = 0, nsecs = 8  
 Reading cluster = 17053, offset = 0, nsecs = 8  
  dfs_closefile: Closing file.... 4019e5c  
  dfs_closehandle: Closed file.... 4019e5cUsing monlib version 2  
 Using version info 2  
   
  dfs_openfile: Opening file....c800-universalk9-mz.SPA.154-3.M4.bin  
  dfs_closefile: Closing file.... 4019e5c  
  dfs_closehandle: Closed file.... 4019e5c  
  dfs_openfile: Opened file /c800-universalk9-mz.SPA.154-3.M4.bin with fib = 4019e5c  
 Reading cluster = 17053, offset = 0, nsecs = 8  
   
 Reading cluster = 17053, offset = 0, nsecs = 128  
 Reading cluster = 17069, offset = 0, nsecs = 128  
 Reading cluster = 17085, offset = 0, nsecs = 128  
 <--- ~1300 lines removed --->  
 Reading cluster = 58703, offset = 0, nsecs = 128  
 Reading cluster = 58719, offset = 0, nsecs = 80  
 IOS Image Load Test  
 ___________________  
 Digitally Signed Production Software  
 Self decompressing the image : ###<snip>### [OK]  

Um. Okay. What's going on here?

It turns out that there are a couple of funny things about debugging filesystem operations:

1. You can't see that it's turned on from within IOS:

C881#debug filesystem flash:
C881#show debugging




C881#



2. The setting persists across both reboots and power interruptions. Apparently this directive sets a hardware flag somewhere. no debug filesystem flash: made those messages go away.

The setting probably applies to any storage device available from within the ROMMON:

 rommon 1 > dev  
 Devices in device table:  
     id name  
   flash: compact flash         
 bootflash: boot flash           
 usbflash0: usbflash0           

This was new to me.

Let's Encrypt plugin for ASA

I wrote a certbot (Let's Encrypt ACME client) plugin for Cisco ASA. It runs on a separate box, talks ACME to the Let's Encrypt service and uses the ASA REST API to manage certificates on the ASA.

Details here.

If you try it out, please let me know how it goes?

Bailed out by Comware's python interpreter

Something funny happened to an IRF stack the other day. The gear was moved to a remote location and, as part of the move, the cables/transceivers were moved around.

The IRF (10Gb/s) and uplink (1Gb/s) ports switched places.

The new IRF ports (5 and 6) got added to the configuration just fine, leaving me with:

 #  
 irf-port 1/1  
  port group interface Ten-GigabitEthernet1/1/1  
  port group interface Ten-GigabitEthernet1/1/2  
  port group interface Ten-GigabitEthernet1/1/5  
  port group interface Ten-GigabitEthernet1/1/6  
 #  
 irf-port 2/2  
  port group interface Ten-GigabitEthernet2/1/1  
  port group interface Ten-GigabitEthernet2/1/2  
  port group interface Ten-GigabitEthernet2/1/5  
  port group interface Ten-GigabitEthernet2/1/6  
 #  

But IRF would't release the old ports making it impossible to repurpose them as uplinks:

 [switch]irf-port 1/1  
 [switch-irf-port1/1]undo port group interface Ten-GigabitEthernet1/1/1  
 Check failed for reason:  
  Can't support IRF on a port with 1000M speed!  
 [switch]

Can't remove a port from the IRF group because it's ineligible to participate in IRF. Okaaaaay...

Without functioning uplinks, it was impossible to transfer the saved configuration away for an off-box edit.

Python to the rescue!
I split the IRF, used python to edit the stored configuration in-place on each IRF member, then rebooted the switches individually. Here's the edit-by-python:

 <switch>python   
 Python 2.7.3 (default, Apr 10 2014, 16:32:11)   
 [GCC 4.4.1] on linux2  
 Type "help", "copyright", "credits" or "license" for more information.  
 >>>   
 >>> import fileinput, re, sys  
 >>> for line in fileinput.input('flash:/startup.cfg', inplace = True):  
 ...  if not re.search(r' port group interface Ten-GigabitEthernet[12]/1/[12]'  
 ,line):  
 ...    sys.stdout.write(line)  
 ...   
 >>> ^D
 <switch> 

Many thanks to my pal Chris Young for both pointing out that I had a python interpreter at my disposal, and suggesting that I use it to butcher the saved configuration.

Amazon Dash Button Events On A Catalyst

Lots of folks are detecting Amazon Dash button events by watching for ARP traffic with python.

I took a slightly different approach by watching for the button's MAC address with an EEM applet.

My Mac 'n Cheese button speaks on the network twice with each push: once right when it's pushed, and then a second time about 40 seconds later.

The applet sleeps for 60 seconds after it's fired to ensure that the button only creates a single event with each press.

 event manager applet macNcheese  
  event mat mac-address 00bb.3a4b.5a01 type add maxrun 90  
  action 1 syslog msg "It's Mac N Cheese time!"  
  action 2 cli command "enable"  
  action 3 cli command "copy https://username:password@some_server/path/to/events.php^V?eventtype=MAC%20N%20CHEESE%20TIME! null:"  
  action 4 wait 60  
  action 5 cli command "clear mac address-table dynamic address 00bb.3a4b.5a01"  

event mat refers to "mac address table" changes. This applet fires only when the button's address is added to the table. Without the add keyword, the event would fire twice, once when the entry is added, and again when the entry is removed from the switch L2 filtering table.

I'm triggering an external event by hitting a web server that's already configured to receive events through HTTP GETs, and I'm using an IOS copy command. It's a bit clunky, but works fine. I could also send SNMP traps, or rely on syslog parsing (Spunk -> StackStorm, perhaps?) to make things happen.

Getting the ^V? characters into the URL string was a little tricky. Typing a ? usually invokes the IOS inline help, so it needs to be escaped by a <ctrl-v>. The event engine will have the same problem typing the ?, so it also needs to type a <ctrl-v>, which I needed to escape in order to type... In the end, I typed three <ctrl-v>s, followed by a ? in order to produce the string above.

This could probably also be done by watching for DHCP snooping events (using the DHCP snooping MIB), but I haven't figured out how to make a DHCP snooping-based applet fire only once for each button press. This is probably worth figuring out because more platforms will support SNMP-based events than mat events.