Beginning in 2016 the Campus Cluster moved away from an instance-based production to a continuous deployment model that allows us to continually add and retire hardware as needed. To see current generation node pricing, visit Buy Compute.


Golub Cluster

Production Date: September 3, 2013

Timelapse video of the cluster installation

The infrastructure of the Golub cluster, is designed to support up to 512 nodes with FDR InfiniBand for applications communications and data transport with a gigabit Ethernet control network. The disk system was selected to support expandability and the GPFS file system.

The details for the hardware components are listed below.

Login Nodes

  • (4) Dell PowerEdge R720 login nodes each configured with:
    • (2) Intel E5-2660 2.2 GHz 8-core processors, 95 W
    • 128 GB RAM via (16) 8 GB 1333 MT/s RDIMMs
    • (2) 300 GB 6G SAS 10K 2.5″ HDD
    • Mellanox ConnectX-3 FDR IB HCA
    • (2) NVIDIA TESLA M2090 GPUs

Compute Nodes (Current): Maximum Count

  • (44) Dell PowerEdge C8000 4U chassis each with:
    • (2) 1400 W power supply units
    • (6) 6 x 120mm high-efficiency fans with PWM control
  • (312) Compute nodes—Dell C8220 compute sleds each with:
    • (2) Intel E5-2670V2 (Ivy Bridge) 2.50 GHz, 25MB cache, 10C, 115 W
    • 64/128/256 GB RAM at customer’s choice
    • (2) 1 TB, 7200 RPM, SATA, 3 Gbps, 2.5″ HDD
    • (4) Intel Ethernet controller i350
  • Compute Node Options:
    • Mellanox ConnectX-3 FDR IB HCA
    • (2) NVIDIA TESLA K40 GPUs

Compute Nodes (Original): Maximum Count

  • (28) Dell PowerEdge C8000 4U chassis each with:
    • (2) 1400 W power supply units
    • (6) 6 x 120mm high-efficiency fans with PWM control
  • (200) Compute nodes—Dell C8220 compute sleds each with:
    • (2) Intel E5-2670 (Sandy Bridge) 2.60 GHz, 20 MB Cache, 8C, 115 W
    • 32/64/128 GB RAM at customer’s choice
    • (2) 1 TB, 7200 RPM, SATA, 3 Gbps, 2.5″ HDD
    • (4) Intel Ethernet Controller i350
  • Compute Node Options:
    • Mellanox ConnectX-3 FDR IB HCA
    • (2) NVIDIA TESLA M2090 GPUs

Network Infrastructure

  • High-speed InfiniBand cluster interconnect
    • Mellanox MSX6518-NR FDR InfiniBand (384-port capable)
  • Management and IPMI control networks
    • (2) Dell PowerConnect 8064F 48-port 10 GigE switches
    • (41) Dell PowerConnect 5548 49-port 1 GigE switches
    • (2) Dell PowerConnect 2848 48-port 1 GigE switches

Rack Infrastructure

  • (9) Netshelter SX 48U 750mm wide, 1200mm deep, model AR3357 racks for compute nodes including (3) high density PDUs with IEC-309 60A plugs
  • (1) Netshelter SX 48U 750mm wide, 1200mm deep, model AR3357 racks for VM hosting, fast data transfer and master nodes including (2) high density PDUs with IEC-309 60A plugs
  • (1) DDN 50U rack for storage subsystem including (6) high density PDUs with IEC-309 60A plugs

Support

  • Basic hardware services: Business hours (5×10) next business day on-site hardware warranty repair
  • Dell hardware limited warranty plus on-site service
  • 24×7 pager support, cross shipment repair replacements for DDN equipment
  • Silver technical support for Mellanox IB fabric
  • 4-year, next-day support on all hardware

Storage Infrastructure

  • (1) DDN SFA12K40D-56IB Couplet with 5 enclosures
  • (60) 3 TB 7200 RPM 2.5″ SATA HDD expandable to 600 HDD
  • (4) Dell PowerEdge R720 GPFS-NDS nodes each configured with:
    • (2) Intel E5-2665 2.4 GHz 8-core processors, 115 W
    • 256 GB RAM via (16) 16 GB 1333 MT/s RDIMMs
    • Mellanox ConnectX-3 dual-port FDR IB HCA
    • Intel X520 DP 10 Gbps DA/SFP+ server adapter
    • (4) 300 GB 15K RPM 6 Gbps SAS HDD

Taub Cluster

Production Date: May 1, 2011

The infrastructure of the Taub cluster—representative of future cluster instances—is designed to support up to 512 nodes with QDR InfiniBand for applications communications and data transport with a gigabit Ethernet control network. The disk system was selected to support expandability and the GPFS file system.

The OS is Scientific Linux 6.1 (Linux 2.6.32).

Admin Nodes

  • (2) DL380 G7 admin nodes configured with:
    • (2) Intel X5650 2.67 GHz 6-core processors, 95 W
    • 24 GB RAM via (12) 2 GB 2Rx8 PC3-10600E-9 DIMMs
    • HP P410 Smart Array Controller with 512 MB BBWC cache
    • (2) HP 146i GB 6G SAS 10K 2.5″ DP ENT HDD
    • HP IB 4x QDR CX-2 PCI-e G2 dual-port HCA
    • N+1 Power

Compute Nodes: Maximum Count

  • (164) HP s6500 4U chassis each with:
    • (4) HP 1200 W CS HE power supply kit
    • (8) HP s6500 redundant fan kit
  • (512) Compute nodes—HP SL390G7 1U servers each configured with:
    • (2) Intel HP X5650 2.66 GHz 6-core processors, 95 W
    • 12/24/48/96 GB RAM at customer’s choice
    • HP 160 GB or 500 GB 3G SATA 7200 RPM 3.5″ QR ETY HDD
    • HP IB enablement kit

Network Infrastructure

  • High-speed Infiniband cluster interconnect
    • (1) Voltaire QDR Infiniband switch configured with 512 ports
    • (1) Voltaire 4036E QDR Infiniband to 10 Gbps Ethernet gateway
  • Management and IPMI control network
    • (16) HP ProCurve 2910al-48G 10/100/1000 Ethernet switches
    • (16) HP ProCurve 2610-48 10/100/1000 Ethernet switches
    • (1) HP ProCurve 5412zl aggregation switch

Rack Infrastructure

  • (1) HP 42U rack containing control nodes and core Ethernet switches including (2) high-voltage PDUs with a total of (2) NEMA L15-30P 30A plugs
  • (8) HP 42U racks for control nodes but only housing leaf Ethernet switches including (16) high-voltage PDUs with a total of (16) IEC-309 60A plugs
  • (1) HP 42U 800mm rack containing Infiniband switches including (2) high-voltage PDUs with a total of (2) NEMA L15-30P 30A plugs

Support

  • Hardware factory integration (AF002A#001, HA453A1-000)
  • Cluster setup and cable labeling (HF482A1) by the Americas Integration Center (AIC)
  • On-site hardware installation (HA124A1#568 and included in HA453A1) provided by Global Deployment Service (GDS)
  • 5-year, next-day support on all hardware

Storage Infrastructure

  • (1) DDN SFA10K IB Couplet with 5 enclosures
  • (80) 2 TB 7200 RPM 3.5″ SATA HDD expandable to 600 HDD
  • (2) DL380 G7 MDS/OSS nodes configured with:
    • (2) Intel E5650 2.67 GHz 6-core processors, 95 W
    • 24 GB RAM via (12) 2 GB 2Rx8 PC3-10600E-9 DIMMs
    • HP P410 Smart Array Controller with 512 MB BBWC cache
    • (2) HP 146 GB 6G SAS 10K 2.5″ DP ENT HDD
    • HP IB 4x QDR CX-2 PCI-e G2 dual-port HCA
    • N+1 Power