Hot Hot Hot Nexus FEXes

Cisco's Nexus 2xxx products are marketed for top-of-rack deployment at the back (hot side) of a server cabinet.

They blow air in the same direction as servers (toward the back).  They're only 1/2 to 2/3 as deep as a typical server.  This combination of features (flow direction and depth) means that the FEX intakes are near the middle of the cabinet.

Airflow management in a typical server cabinet includes blanking panels covering unused server positions and baffles between the vertical mounting rails and the sides of the cabinet.  As a result, the entire cabinet runs hot, except for the couple of inches between the front door and the server air intake.

Fabric extenders inevitably run hot.  I wish Cisco had made the FEXes full-depth so that they'd always breathe in cool air.

The statistics below are from an environment that's not quite as severe as my drawing indicates.  In fact, none of the baffles or blanking panels are in place.  These FEXes aren't getting cool air because hot air is actually exhausting on the cold side of the cabinet near the FEX intake.  Hot server exhaust is circling from the rear of the cabinet, up the sides and out through the area of the FEX intake.  The FEXes don't move enough air to overcome the high pressure exhaust from the servers.

-----------------------------------------------------------------

Module   Sensor     MajorThresh   MinorThres   CurTemp     Status
                    (Celsius)     (Celsius)    (Celsius)
-----------------------------------------------------------------

1        Outlet-1   60            50           48          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           46          ok
1        Inlet-1    50            40           41          minor alarm
1        Outlet-1   60            50           49          ok
1        Inlet-1    50            40           43          minor alarm
1        Outlet-1   60            50           48          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           41          ok
1        Inlet-1    50            40           33          ok
1        Outlet-1   60            50           39          ok
1        Inlet-1    50            40           33          ok
1        Outlet-1   60            50           47          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           44          ok
1        Inlet-1    50            40           38          ok
1        Outlet-1   60            50           50          ok
1        Inlet-1    50            40           44          minor alarm
1        Outlet-1   60            50           48          ok
1        Inlet-1    50            40           42          minor alarm

1        Outlet-1   60            50           50          ok
1        Inlet-1    50            40           43          minor alarm
1        Outlet-1   60            50           49          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           49          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           47          ok
1        Inlet-1    50            40           40          minor alarm
1        Outlet-1   60            50           41          ok
1        Inlet-1    50            40           34          ok
1        Outlet-1   60            50           40          ok
1        Inlet-1    50            40           34          ok
1        Outlet-1   60            50           48          ok
1        Inlet-1    50            40           42          minor alarm
1        Outlet-1   60            50           47          ok
1        Inlet-1    50            40           41          minor alarm
1        Outlet-1   60            50           47          ok
1        Inlet-1    50            40           41          minor alarm
1        Outlet-1   60            50           45          ok
1        Inlet-1    50            40           39          ok

Omitting the blanking panels obviously doesn't help, nor does mounting full-depth equipment directly under the FEXes.  The only solution that makes sense to me is to truly isolate the FEX intake from the server exhaust.  But how can this be accomplished?

A rack-mountable air duct at the front of the rack (in place of the top blanking panel) would be great, but I don't know if such a product exists.  [It exists - see update 2 below]  Until then, it might come down to duct-taping paper towel tubes to the FEX intakes  :-)

Moving the blanking panels and side baffles to the rear of the cabinet might do the trick, but...

• It only works when dealing with equipment of a uniform depth.  Alternating short/long servers in a heterogeneous server environment make the rear blanking panel strategy fall apart.
• I've never seen a rear-blanked server cabinet, nor have any of the facilities guys I've talked to, and they're a conservative bunch, not interested in going out on a limb.

Have you seen this problem in your data center?

What are you doing about it?

Update: TAC has pointed me to some documentation that addresses my airflow concerns.  The answer seems completely outrageous:

Apparently all of my FEXes are mounted wrong.  The documentation shows the fixed rack ear (1) mounted to the intake end of the FEX.  This would mount the FEX flush with the cool side of the cabinet, making all of its switchports completely inaccessible in a full cabinet.

The relevant section of the Cisco on Cisco tour of the Richardson data center doesn't play anymore, but I've seen those videos, and I'm pretty sure that Cisco's own FEXes are mounted like the ones in my drawing at the top of this post, contrary to the documentation and the drawing above.

Wondering how I missed this detail at install time, I cracked open the carton of a factory-fresh 2+ year old 2148T.  The box contained the FEX, mounting hardware, and a safety sheet.  No install guide.  No rackmount drawings.  No documentation DVD.

If you've ever seen a FEX mounted flush with the cold side of the cabient, please leave a comment about it.  How do you reach the uplink and downlink ports?

Update 2:  The best solution so far is the Panduit CDE2 air duct.  It's made to be compatible with the 4948E and Nexus 2xxx, and is even referenced by Cisco's 4948E documentation.  This product appears to be a sheet metal duct which extends the air intake to the cold side of the cabinet.  It doubles the FEX space requirements to 2U.  It's frustrating that this accessory is required to operate a FEX in the typical deployment scenario, but I'm glad it's available!