Computing cluster

HPC cluster is based on blade systems from IBM and Dell. Dell blade servers were purchased within VAVINET project, blade servers from IBM within projects of FR CESNET and FRVŠ. Together there is now 102 modules, each with two 4 to 28 core CPUs and 8 to 256GB of RAM. On all blade servers we use 64bit CentOS operating system. To store large datasets there are 12 storage servers with total capacity of over 500TB.

All computations on the cluster are maintained by the Sun Grid Engine (SGE) job management system. The tasks have to be described with a script (usually a shell script) and this script is submitted to SGE. Based on current state of computational nodes as well as storage and other conditions SGE decides when and on which node to assign each task. Maximal theoretic size of a parallel computation is dynamic, depends on the current state of all nodes and currently it could be as high as 2400 processes.

Blade servers

Chassis	Modules	Hardware
blademm1	blade001 - blade012	IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW
blademm1	blade001 - blade012	12 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) - blade001-005,009,010,012 RAM DDR3 30GB (3 channels) - blade006-008,011 HDD SAS 2,5" 146 GB
blademm2	blade013 - blade024	IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW
blademm2	blade013 - blade024	12 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) HDD SAS 2,5" 146 GB - blade013-018 HDD SAS 2,5" 300 GB - blade019-024
blademm3	blade025 - blade029	IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW
	blade025 - blade029	5 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) HDD SAS 2,5" 300 GB
	blade030 - blade036	7 modules HS21XM (7995) 2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 20GB (2 channels) HDD SAS 2,5" 300 GB
blademm4	blade037 - blade048	IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW
blademm4	blade037 - blade048	12 modules HS21XM (7995) 2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 8 GB (2 channels) HDD SCSI 73 GB
dellmm1	blade055 - blade070	Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
		6 modules Blade Dell M630 2 CPU Intel Xeon E5-2680 v3 2,4 GHz, 14/28core, 35 MB cache RAM DDR4-2133 128 GB (4 channels) HDD 2,5" 300 GB SAS 15k LAN Broadcom 57810-k 2 x 10Gb/s KR
		10 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR
dellmm2	blade071 - blade086	Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
dellmm2	blade071 - blade086	16 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR
dellmm3	blade087 - blade102	Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
		14 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64/128 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR
		2 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 256 GB HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR

HPC servers

welda1, welda2, ulrika, svatava

Supermicro SC113TQ
MB Supermicro X10DRW-E
CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache
RAM DDR4-2133 32GB (4 channels)
HDD Crucial SSD 250 GB
PSU 600W

GPU servers

supergpu1,supergpu2	Supermicro 7047GR-TPRF MB Supermicro X9DRG-QF 2 CPU Intel Xeon E5-2630v2 2,6 GHz, 6/12core, 15 MB cache RAM DDR3-1600 64GB (4 channels) HDD WD RE4 500GB Ethernet 10 Gb/s Intel X520-DA2 4 GPU NVidia GTX 980 4GB GDDR5 PSU 2x1600 W
supergpu3,supergpu4	Supermicro 7048GR-TR MB Supermicro X10DRG-Q 2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache RAM DDR4-2133 64GB (4 channels) SSD Crucial 250GB Ethernet 1 Gb/s 4 GPU NVidia GTX 1080 8GB GDDR5 PSU 2x2000 W
superphi1,superphi2	Supermicro 7048GR-TR MB Supermicro X10DRG-Q 2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache RAM DDR4-2133 64GB (4 channels) SSD Crucial 250GB Ethernet 1 Gb/s 1 GPU NVidia GTX 980 4GB GDDR5 3 Intel Xeon Phi PSU 2x2000 W
dellgpu1,dellgpu2	Dell T620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64GB (4 channels) HDD WD RE4 500GB LAN Broadcom 57810S 2 x 10Gb/s SFP+ 4 GPU NVidia GTX 980 4GB GDDR5 PSU 2x1100 W

File servers

matylda1-6	Supermicro SC113TQ MB Supermicro X10DRW-E 2 CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache RAM DDR4-2133 64GB HDD Intel SSD 80 GB Ethernet 10 Gb/s Intel X520-DA2 72 TB RAID-6 3Ware (LSI) 9750SA-8e External JBOD Supermicro SC836E16-R1200B
scratch1-6	Supermicro SC815TQ MB Supermicro X9DRW-iF CPU Intel Xeon E5-2630, 2,3 GHz, 6/12core, 15 MB cache RAM DDR3-1600 16 GB (4 channels) Ethernet 10 Gb/s Intel X520-DA2 10 TB RAID-6 3Ware (LSI) 9750SA-8e External JBOD Supermicro SC826E16-R500LP

Network interconnect 10Gb/s

cluster switches

Extreme Networks Summit X650-24x
4 x 24 ports Ethernet 10 Gb/s SFP+
SummitStack 256 Gb/s ring

HPC switch2 traffic

HPC switch3 traffic

Nacházíte se v sekcích