Nexus 5K Layout for 10Gb/s Servers - Part 2

My first couple of blog posts were about building 1Gb/s top of rack switching using the Nexus 5000 product line.  This is a new series comparing some options for 10Gb/s top of rack switching with Nexus 5500 switches.

These posts assume a brand-new deployment and attempt to cover all of the bits required to implement the solution.  I'm including everything from the core layer's SFP+ downlink modules through the SFP+ ports for server access.  I'm assuming that the core consists of SFP+ interfaces capable of supporting twinax cables, and that each server will provide it's own twinax cable of the appropriate length.
Each scenario supports racks housing sixteen 2U servers.  Each server has a pair of SFP+ based 10Gb/s NICs and a single 100Mb/s iLO interface.

Option 2 - Top of Rack Nexus 5548
This is a 2-rack pod consisting of two Nexus 5548 switches and two Catalyst 2960s for iLO/PDU/environmental/etc...

Option 2

32 servers have 640Gb/s of access bandwidth with no oversubscription inside the pod.  The pod's uplink is oversubscribed by 4:1.

The 5548 management connections (Ethernet and serial) will require the installation of some structured wiring (copper) so that they can reach their management network and terminal server.

The 2960 console connections are less critical.  My rule on this is: If the clients I'm supporting can't be bothered to provision more than a single NIC, then they can't be all that important.  Redundancy and supportability considerations at the network layer may be compromised.

Here's the resulting bill of matierals with pricing:

Config Set Lines
Line	Item / Part#/Description		List Price	Qty.	Discount(s)	Unit Price	Extended Price
1.0	N5K-C5548UP-FA Nexus 5548 UP Chassis, 32 10GbE Ports, 2 PS, 2 Fans		25,600.00	2	0%	25,600.00	51,200.00
1.1	N5548P-FAN Nexus 5548P and 5548UP Fan Module, Front to Back Airflow		0.00	4	0%	0.00	0.00
1.2	N55-PAC-750W Nexus 5500 PS, 750W, Front to Back Airflow(Port-Side Outlet)		0.00	4	0%	0.00	0.00
1.3	CAB-C13-CBN Cabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors		0.00	4	0%	0.00	0.00
1.4	GLC-T 1000BASE-T SFP		395.00	4	0%	395.00	1,580.00
1.5	SFP-10G-SR 10GBASE-SR SFP Module		1,495.00	32	0%	1,495.00	47,840.00
1.6	SFP-H10GB-CU1M 10GBASE-CU SFP+ Cable 1 Meter		150.00	6	0%	150.00	900.00
1.7	N55-DL2 Nexus 5548 Layer 2 Daughter Card		0.00	2	0%	0.00	0.00
1.8	N55-M16UP Nexus 5500 Unified Mod 16p 10GE Eth/FCoE OR 16p 8/4/2/1G FC		11,200.00	2	0%	11,200.00	22,400.00
1.9	N5KUK9-503N2.1 Nexus 5000 Base OS Software Rel 5.0(3)N2(1)		0.00	2	0%	0.00	0.00
1.10	DCNM-L-NXACCK9 DCNM for LAN Advanced Edition for N1/2/4/5K		0.00	2	0%	0.00	0.00
1.11	N5548-ACC-KIT Nexus 5548 Chassis Accessory Kit		0.00	2	0%	0.00	0.00
2.0	WS-C2960-24TC-S Catalyst 2960 24 10/100 + 2 T/SFP LAN Lite Image		725.00	2	0%	725.00	1,450.00

List price for this layout is $125,370, or $62,685 for each 16-server cabinet.

The 5548s are maxed-out.  All of the other configurations use an 80Gb/s vPC peer-link, but these guys can only accomodate 40Gb/s, which is probably fine.  Moving the vPC peer-keepalive traffic and Catalyst 2960 uplinks onto some other network would make it possible to max out the peer-link, but I don't believe it matters.