These posts assume a brand-new deployment and attempt to cover all of the bits required to implement the solution. I'm including everything from the core layer's SFP+ downlink modules through the SFP+ ports for server access. I'm assuming that the core consists of SFP+ interfaces capable of supporting twinax cables, and that each server will provide it's own twinax cable of the appropriate length.
Each scenario supports racks housing sixteen 2U servers. Each server has a pair of SFP+ based 10Gb/s NICs and a single 100Mb/s iLO interface.
Option 4 - Top of Rack Nexus 2232 + Catalyst 2960
This is a 3-rack pod consisting of two Nexus 5548 switches in a central location near the core, and Nexus 2232 and Catalyst 2960 deployed at the top of rack.
 |
| Option 4 |
Each Nexus 2232 uplinks to a single Nexus 5548, and the Catalyst 2960s are uplinked via vPC to the fabric extenders. Connecting the 2960 to the FEX requires some special consideration. In this case, there are two reasonably safe ways to do it:
- Configure BPDUfilter on the uplink EtherChannel and BPDUguard on all the field ports. Put each 2960 into its own VLAN. Any cable mistakenly linked between racks should be killed off by BPDUguard immediately, and the 2960s can't form a loop because they're in different VLANs anyway.
- Configure the lanbase-routing feature on the 2960 and run each 2960 uplink as a /31 routed link. BPDUfilter will still be required on the 2960 (routed interfaces aren't supported by lanbase-routing so you have to use SVI+VLAN), but a loop cannot form because each 2960 (uplink and downlink) is in a different VLAN. Might not be possible with this specific model of 2960. Upgrading the 2960 to a WS-C3560V2-24TS-S ($3K for limited L3 features).
48 servers have 960Gb/s of access bandwidth with 2:1 oversubscription inside the pod. The pod's uplink is oversubscribed by 6:1, same as Option 1 and option 3.
Because the Nexus 5548 are installed in a central location (not in the server row), management connections (Ethernet and serial) do not require any special consideration. Only multimode fiber needs to be installed into the server row.
The 2960 console connections are less critical. My rule on this is: If the clients I'm supporting can't be bothered to provision more than a single NIC, then they can't be all that important. Redundancy and supportability considerations at the network layer may be compromised.
The advantages of this configuration include:
- Plenty of capacity for adding servers, because each 10Gb/s FEX is only half-full (oversubscription would obviously increase).
- Use of inexpensive twinax connections for 7K<-->5K links. There are more boxes here, but the overall price is lower becasue of this change.
- Centralized switch management - serial and Ethernet management links are all in one place.
- This model translates directly to 10GBASE-T switching. When servers begin shipping with onboard 10GBASE-T interfaces, we switch from Nexus 2232PP to 2232TM, and the architecture continues to work. This isn't possible with top-of-rack Nexus 5500s right now.
Basically, it's the same topology as option 3, but I've swapped out the 2224s in favor of 2960s to save a few bucks. Spending $25,000 for 100Mb/s iLO ports drove me a little crazy.
Here's the resulting bill of matierals with pricing:
List price for this layout is $152,205, or $50,735 for each 16-server cabinet.
Here's the resulting bill of matierals with pricing:
| Config Set Lines | |||||||
| Line | Item / Part#/Description | List Price | Qty. | Discount(s) | Unit Price | Extended Price | |
|---|---|---|---|---|---|---|---|
| 1.0 | N5548UPM-4N2232PF Nexus 5548UP/Expansion Module/4xN2232PP/64xFET | 78,000.00 | 1 | 0% | 78,000.00 | 78,000.00 | |
| 1.1 | N55-DL2 Nexus 5548 Layer 2 Daughter Card | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.2 | N55-M16UP-B Nexus 5500 Series Module 16p Unified | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.3 | N5KUK9-503N2.1 Nexus 5000 Base OS Software Rel 5.0(3)N2(1) | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.4 | DCNM-L-NXACCK9 DCNM for LAN Advanced Edition for N1/2/4/5K | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.5 | GLC-T 1000BASE-T SFP | 395.00 | 3 | 0% | 395.00 | 1,185.00 | |
| 1.6 | SFP-H10GB-CU1M 10GBASE-CU SFP+ Cable 1 Meter | 150.00 | 5 | 0% | 150.00 | 750.00 | |
| 1.7 | SFP-H10GB-CU3M 10GBASE-CU SFP+ Cable 3 Meter | 210.00 | 8 | 0% | 210.00 | 1,680.00 | |
| 1.8 | N5548P-FAN Nexus 5548P and 5548UP Fan Module, Front to Back Airflow | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.9 | N55-PAC-750W Nexus 5500 PS, 750W, Front to Back Airflow(Port-Side Outlet) | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.10 | CAB-C13-CBN Cabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 1.11 | CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length | 0.00 | 8 | 0% | 0.00 | 0.00 | |
| 1.12 | N2K-C2232PP-BUN Standard airflow/AC pack: N2K-C2232PP-10GE, 2AC PS, 1Fan | 0.00 | 4 | 0% | 0.00 | 0.00 | |
| 1.13 | N5K-C5548UP-BUN Nexus 5548UP in N5548UP-N2K Bundle | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 1.14 | FET-10G 10G Line Extender for FEX | 0.00 | 64 | 0% | 0.00 | 0.00 | |
| 1.15 | N5548-ACC-KIT Nexus 5548 Chassis Accessory Kit | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.0 | N5K-C5548UP-FA Nexus 5548 UP Chassis, 32 10GbE Ports, 2 PS, 2 Fans | 25,600.00 | 1 | 0% | 25,600.00 | 25,600.00 | |
| 2.1 | N5548P-FAN Nexus 5548P and 5548UP Fan Module, Front to Back Airflow | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.2 | N55-PAC-750W Nexus 5500 PS, 750W, Front to Back Airflow(Port-Side Outlet) | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.3 | CAB-C13-CBN Cabinet Jumper Power Cord, 250 VAC 10A, C14-C13 Connectors | 0.00 | 2 | 0% | 0.00 | 0.00 | |
| 2.4 | GLC-T 1000BASE-T SFP | 395.00 | 3 | 0% | 395.00 | 1,185.00 | |
| 2.5 | SFP-H10GB-CU1M 10GBASE-CU SFP+ Cable 1 Meter | 150.00 | 5 | 0% | 150.00 | 750.00 | |
| 2.6 | SFP-H10GB-CU3M 10GBASE-CU SFP+ Cable 3 Meter | 210.00 | 8 | 0% | 210.00 | 1,680.00 | |
| 2.7 | N55-DL2 Nexus 5548 Layer 2 Daughter Card | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.8 | N55-M16UP Nexus 5500 Unified Mod 16p 10GE Eth/FCoE OR 16p 8/4/2/1G FC | 11,200.00 | 1 | 0% | 11,200.00 | 11,200.00 | |
| 2.9 | N5KUK9-503N2.1 Nexus 5000 Base OS Software Rel 5.0(3)N2(1) | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.10 | DCNM-L-NXACCK9 DCNM for LAN Advanced Edition for N1/2/4/5K | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 2.11 | N5548-ACC-KIT Nexus 5548 Chassis Accessory Kit | 0.00 | 1 | 0% | 0.00 | 0.00 | |
| 3.0 | N2K-C2232PF-10GE Nexus 2232PP with 16 FET (2 AC PS, 1 FAN (Std Airflow)) | 14,000.00 | 2 | 0% | 14,000.00 | 28,000.00 | |
| 3.1 | CAB-C13-C14-2M Power Cord Jumper, C13-C14 Connectors, 2 Meter Length | 0.00 | 4 | 0% | 0.00 | 0.00 | |
| 3.2 | FET-10G 10G Line Extender for FEX | 0.00 | 32 | 0% | 0.00 | 0.00 | |
| 4.0 | WS-C2960-24TC-S Catalyst 2960 24 10/100 + 2 T/SFP LAN Lite Image | 725.00 | 3 | 0% | 725.00 | 2,175.00 | |
| 4.1 | CAB-C13-C14-AC Power cord, C13 to C14 (recessed receptacle), 10A | 0.00 | 3 | 0% | 0.00 | 0.00 | |
List price for this layout is $152,205, or $50,735 for each 16-server cabinet.
How are you channeling to the 2960 with the FEXes dual-homed?
ReplyDeleteSorry, one more. :x
ReplyDeleteAre routed interfaces now supported on the 2960s? I could be mistaken, but when routing was originally released on the 2960s, I believe it was via SVIs only.
Hey Colby,
ReplyDeleteThese FEXes are single homed. Look closely and you'll see. Maybe I should make them more transparent. The only vPC-attached FEXes appear in option #3.
vPC FEXes with vPC hosts is coming soon, from what I understand!
Routed interfaces are not supported on the 2960 as far as I know. I tried to indicate that they'd be SVIs -- that's what I meant when I mentioned BPDUfilter and routed interfaces not supported... But I see that it was confusing. I'll change the language to make it make more sense.
Thanks!
Ah, duh, I see it. You got my hopes up. Supposedly they're fixing that limitation soon.
ReplyDeletePs. Check my post on NF.:p
I came up with an alternate version of this design, including BOM posted here.
ReplyDelete