These posts assume a brand-new deployment and attempt to cover all of the bits required to implement the solution. I'm including everything from the core layer's SFP+ downlink modules through the SFP+ ports for server access. I'm assuming that the core consists of SFP+ interfaces capable of supporting twinax cables, and that each server will provide it's own twinax cable of the appropriate length.
Each scenario supports racks housing sixteen 2U servers. Each server has a pair of SFP+ based 10Gb/s NICs and a single 100Mb/s iLO interface.
Option 3 - Top of Rack Nexus 2232
This is a 3-rack pod consisting of two Nexus 5548 switches in a central location near the core, and Nexus 2232 and 2224 deployed at the top of rack.
 |
| Option 3 |
48 servers have 960Gb/s of access bandwidth with 2:1 oversubscription inside the pod. The pod's uplink is oversubscribed by 6:1, same as Option 1.
Because the Nexus 5548 are installed in a central location (not in the server row), management connections (Ethernet and serial) do not require any special consideration. Only multimode fiber needs to be installed into the server row.
The advantages of this configuration include:
- Plenty of capacity for adding servers, because each 10Gb/s FEX is only half-full (oversubscription would obviously increase)
- No iLO switches to manage - switching for iLO/PDU/etc... is handled by the Nexus now.
- Use of inexpensive twinax connections for 7K<-->5K links. There are more boxes here, but the overall price is lower becasue of this change.
- Centralized switch management - serial and Ethernet mangement links are all in one place.
- This model translates directly to 10GBASE-T switching. When servers begin shipping with onboard 10GBASE-T interfaces, we switch from Nexus 2232PP to 2232TM, and the architecture continues to work. That isn't possible with top-of-rack Nexus 5500s right now.
Here's the resulting bill of matierals with pricing:
| Config Set Lines | |||||||
| Line | Item / Part#/Description | List Price | Qty. | Discount(s) | Unit Price | Extended Price | |
|---|---|---|---|---|---|---|---|
| 1.0 | N5548UPM-4N2232PF Nexus 5548UP/Expansion Module/4xN2232PP/64xFET | 78,000.00 | 1 | 0% | 78,000.00 | 78,000.00 | |
| 1.1 | N55-DL2 Nexus 5548 Layer 2 Daughter Card | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.2 | N55-M16UP-B Nexus 5500 Series Module 16p Unified | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.3 | N5KUK9-503N2.1 Nexus 5000 Base OS Software Rel 5.0(3)N2(1) | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.4 | DCNM-L-NXACCK9 DCNM for LAN Advanced Edition for N1/2/4/5K | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.5 | SFP-H10GB-CU1M 10GBASE-CU SFP+ Cable 1 Meter | 150.00 | 10 | 0% | 150.00 | 1,500.00 | |
| 1.6 | N5548P-FAN Nexus 5548P and 5548UP Fan Module, Front to Back Airflow | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.7 | N55-PAC-750W Nexus 5500 PS, 750W, Front to Back Airflow(Port-Side Outlet) | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.8 | CAB-C13-CBN Cabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.9 | CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length | 0.00 | 8 | 0% | 0.00 | 0.00 | |
| 1.10 | N2K-C2232PP-BUN Standard airflow/AC pack: N2K-C2232PP-10GE, 2AC PS, 1Fan | 0.00 | 4 | 0% | 0.00 | 0.00 | |
| 1.11 | N5K-C5548UP-BUN Nexus 5548UP in N5548UP-N2K Bundle | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.12 | FET-10G 10G Line Extender for FEX | 0.00 | 64 | 0% | 0.00 | 0.00 | |
| 1.13 | N5548-ACC-KIT Nexus 5548 Chassis Accessory Kit | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.0 | N5K-C5548UP-FA Nexus 5548 UP Chassis, 32 10GbE Ports, 2 PS, 2 Fans | 25,600.00 | 1 | 0% | 25,600.00 | 25,600.00 | |
| 2.1 | N5548P-FAN Nexus 5548P and 5548UP Fan Module, Front to Back Airflow | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.2 | N55-PAC-750W Nexus 5500 PS, 750W, Front to Back Airflow(Port-Side Outlet) | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.3 | CAB-C13-CBN Cabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.4 | N55-DL2 Nexus 5548 Layer 2 Daughter Card | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.5 | N55-M16UP Nexus 5500 Unified Mod 16p 10GE Eth/FCoE OR 16p 8/4/2/1G FC | 11,200.00 | 1 | 0% | 11,200.00 | 11,200.00 | |
| 2.6 | N5KUK9-503N2.1 Nexus 5000 Base OS Software Rel 5.0(3)N2(1) | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.7 | DCNM-L-NXACCK9 DCNM for LAN Advanced Edition for N1/2/4/5K | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.8 | N5548-ACC-KIT Nexus 5548 Chassis Accessory Kit | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 3.0 | N2K-C2232PF-10GE Nexus 2232PP with 16 FET (2 AC PS, 1 FAN (Std Airflow)) | 14,000.00 | 2 | 0% | 14,000.00 | 28,000.00 | |
| 3.1 | CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length | 0.00 | 4 | 0% | 0.00 | 0.00 | |
| 3.2 | FET-10G 10G Line Extender for FEX | 0.00 | 32 | 0% | 0.00 | 0.00 | |
| 4.0 | N2K-C2224TF Nexus 2224TP with 4 FET, choice of airflow/power | 8,000.00 | 3 | 0% | 8,000.00 | 24,000.00 | |
| 4.1 | CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length | 0.00 | 6 | 0% | 0.00 | 0.00 | |
| 4.2 | N2224TP-FA-BUN Standard airflow pack: N2K-C2224TP-1GE, 2AC PS, 1Fan | 0.00 | 3 | 0% | 0.00 | 0.00 | |
| 4.3 | FET-10G 10G Line Extender for FEX | 0.00 | 12 | 0% | 0.00 | 0.00 | |
List price for this layout is $168,300, or $56,100 for each 16-server cabinet.
When you say ~$56k for each 16-server cabinet, are you adding the cost of the 5k's each time? I don't think that would be accurate, as you buy the 5k's once, and when you add another cabinet, your only adding the 2k's and cabling. Am I missing something?
ReplyDeleteThat $56K is 1/3 of the cost of the build presented here. With this topology you can't have a 4th cabinet because the 5548 is full:
ReplyDelete- 8 uplinks
- 8 crosslinks
- 24 2232 downlinks
- 3 2224 downlinks
- 2 vpc keepalive links
45 of 48 ports are used.
The 4th-6th cabinets require you to buy the whole $168K pod again. At $56K per cabinet.
The per-cabinet cost is only useful when comparing different designs against one another because they might drive you to different pod sizes.