Wednesday, December 28, 2011

Powering Network Gear | NetworkingNerd Follow-Up

Tom Hollingsworth published a great post about some of the common NEMA and IEC power connectors that you're likely to encounter when working on network gear in North America. Tom's post inspired me to throw together a short list of gotchas about powering network gear.

North America Power Cord
One of the many power cord choices you'll encounter when ordering a Cisco switch is this one:

CAB-N5K6A-NAPower Cord, 210/220V 30A North America

This is a post about providing power in North America, so it sounds like a good cable, right? Wrong. The male end of this cable features a NEMA 6-15 plug. It would have been nice of Cisco to mention that detail in the product description, huh? Also, I'm a little puzzled by the "30A" reference in the description. A NEMA 6-15 outlet should be backed by a 15A circuit breaker, and the C13 connector is rated for only 10A, as the sketch below indicates.

CAB-N5K6A-NA with NEMA 6-15 connector.
Drawing copied from Nexus 5000 Install Guide.
It's such a weird cable that Tom didn't bother to mention it in his cable rundown post, and I don't think I've ever seen one of these outlets in the wild. I have, however, seen a customer place a large order, and specify these cables (two of them, actually!) for every top-of-rack component.

Cable Length and Where Is The Inlet?
Consider the following power cord choices for the Nexus 5020 switch (note that this is just 1/3 of the choices for this platform):

CAB-N5K6A-NAPower Cord, 210/220V 30A North America
CAB-AC-250V/13ANorth America,NEMA L6-20 250V/20A plug-IEC320/C13 receptacle
CAB-C13-C14-2MPower Cord Jumper, C13-C14 Connectors, 2 Meter Length
CAB-9K12A-NAPower Cord, 125VAC 13A NEMA 5-15 Plug, North America
CAB-C13-CBNCabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors

It's pretty common to find power strips with IEC C14 outlets in server racks these days, and the Nexus 5000 has C13 power inlets. A C13-C14 cable seems like the obvious choice. But there are two of them listed here! That's because CAB-C13-C14-2M is 2m long (as indicated), and the CAB-C13-CBN is only .7m long. Again, it would have been nice of Cisco to mention this detail in the product description.

For a Top-Of-Rack Nexus 5010/5020, CAB-C13-CBN is probably the right choice. The power inlets are on the back (hot aisle side) of the switch, right where we're probably going to find a power outlet. 2m would be way too much power cord for this application.

But what about a Top-Of-Rack Nexus 5500, Nexus 2000, or fixed-configuration Catalyst switch? Those units tend to have the power inlet on the opposite side. A .7m cable would be too short, so we should order the CAB-C13-C14-2M for those.

What Voltage?
When talking about NEMA outlets, it's easy to know what voltage you're going to find. NEMA 5-xxx indicates 110V, and NEMA 6-xxx indicates 220V. But the IEC outlets can be unpredictable. They're probably going to be 220V, in spite of the fact that the C13 cords powering the system I'm sitting in front of right now deliver only 110V.

Most data center gear includes auto-ranging power supplies, so it's no problem, but things that weren't intended to live in the data center might have a voltage selector switch. I've seen Sun and Dell workstations blown up by connecting them to 220V power without setting the manual voltage selector switch.

Along those same lines, I always carry one of these adapter cords so that I can charge my laptop in the data center.
It's super-handy, but creates a situation where you might accidentally plug a 110V device into 220V power. Be careful.

Power Supplies: Bigger Isn't Better
Many of my customers will automatically select the biggest available power supply when specifying their chassis-based switches. Hey, bigger is better, and they only cost a little bit more, right?

Large power supplies generally get big power through one of two options:
  • Multiple power inputs
  • High current power inputs
The high current power inlet approach can be a problem. The largest removable power cords we have available are the C19 connector type, rated for up to 16A. Power supplies with 30A inlets like the 4000W unit for Catalyst 6500 and the 7500W unit for Nexus 7000 have fixed cords that can't be disconnected from the power supply.

4KW PSU for Catalyst 6500
7.5KW PSU for Nexus 7000

When most power supplies fail, you simply unplug the cable from the PSU, replace the PSU, and reattach the cable. With fixed cords, you'll need to fish the whole cord, along with it's huge connector out of the rack. Sometimes pulling that cord out of the rack isn't possible because too many new cables have been run through the penetration in the floor/rack/whatever since the switch was installed. Now what? Chopping off the end of the dead cord is a reasonable option, but how do you get the new PSU installed?

Power supplies with removable cords are often preferable because they simplify operations.

2 PDUs; 2 PSUs; 4 cords -- Now what?
Unfortunately, multiple input power supplies can complicate the initial design. Cisco refers to this set of issues using the phrases "input source redundancy" and "power supply redunancy." The issue boils down to how will you mesh up the four power cables between two PSUs and two PDUs?

You probably want to run both power cords from each PSU to the same circuit/PDU (assuming the PDU can deliver the required current), but people tend to split each device between multiple PDUs. If/when a PDU fails, suddenly each power supply will drop to half of it's previous capacity. If you've configured the switch for full power redundancy mode, then the switch will begin shutting down line cards until the allocated power falls below the threshold represented by half of one power supply. It's ugly, and I've seen it happen more than once.

The same thing comes up when configuring 3750X stackable switches with StackPower. Think carefully about how the power cords align with available circuits. Assume you're going to lose a circuit. Will the stack survive? If you configure every switch with dual power supplies, then it'll be fine, but you might as well skip StackPower at that point because you've just made each switch individually redundant.

To solve this problem the StackPower way, then you need to make sure that the stack power pool has sufficient capacity even when you're down a circuit. Unfortunately, there's no way to validate power cable to circuit mapping remotely. The only way to make sure that this is done correctly is to visit the closet and trace out the power cords.

2 comments:

  1. Timely post! I just got burned by the CAB-N5K6A-NA cable. Luckily it was only for a half dozen 2248's, so it wasn't a massive amount of replacement power cords needed. Unfortunately, this is the first choice in the Cisco configuration tool and the correct plug is a bit further down the list.

    I enjoy reading your blog!

    Matthew Norwood

    ReplyDelete
  2. Hey Matthew, thanks for the compliment. I enjoy your blog as well. I'm glad you and family had a nice time at Disney!

    I'm sorry the post wasn't *more* timely. I blame @NetworkingNerd for not inspiring it soon enough :-)

    ReplyDelete