Computing cluster

HPC cluster is based on blade systems from IBM and Dell. Dell blade servers were purchased within VAVINET project, blade servers from IBM within projects of FR CESNET and FRVŠ. Together there is now 102 modules, each with two 4 to 28 core CPUs and 8 to 256GB of RAM. On all blade servers we use 64bit CentOS operating system. To store large datasets there are 12 storage servers with total capacity of over 500TB.

All computations on the cluster are maintained by the Sun Grid Engine (SGE) job management system. The tasks have to be described with a script (usually a shell script) and this script is submitted to SGE. Based on current state of computational nodes as well as storage and other conditions SGE decides when and on which node to assign each task. Maximal theoretic size of a parallel computation is dynamic, depends on the current state of all nodes and currently it could be as high as 2400 processes.

Blade servers

ChassisModulesHardware

blademm1

blade001 - blade012
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s
PSU 4 x 2kW
12 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels) - blade001-005,009,010,012
RAM DDR3 30GB (3 channels) - blade006-008,011
HDD SAS 2,5" 146 GB

blademm2

blade013 - blade024
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s
PSU 4 x 2kW
12 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels)
HDD SAS 2,5" 146 GB - blade013-018
HDD SAS 2,5" 300 GB - blade019-024

blademm3

blade025 - blade029
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s
PSU 4 x 2kW
5 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels)
HDD SAS 2,5" 300 GB

blade030 - blade036
7 modules HS21XM (7995)
2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 20GB (2 channels)
HDD SAS 2,5" 300 GB

blademm4

blade037 - blade048
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s
PSU 4 x 2kW
12 modules HS21XM (7995)
2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 8 GB (2 channels)
HDD SCSI 73 GB

dellmm1

blade055 - blade070
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
6 modules Blade Dell M630
2 CPU Intel Xeon E5-2680 v3 2,4 GHz, 14/28core, 35 MB cache
RAM DDR4-2133 128 GB (4 channels)
HDD 2,5" 300 GB SAS 15k
LAN Broadcom 57810-k 2 x 10Gb/s KR
10 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR

dellmm2

blade071 - blade086
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
16 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR

dellmm3

blade087 - blade102
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s
14 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64/128 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR
2 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 256 GB
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR

HPC servers


welda1, welda2, ulrika, svatava
Supermicro SC113TQ
MB Supermicro X10DRW-E
CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache
RAM DDR4-2133 32GB (4 channels)
HDD Crucial SSD 250 GB
PSU 600W

GPU servers


supergpu1,supergpu2
Supermicro 7047GR-TPRF
MB Supermicro X9DRG-QF
2 CPU Intel Xeon E5-2630v2 2,6 GHz, 6/12core, 15 MB cache
RAM DDR3-1600 64GB (4 channels)
HDD WD RE4 500GB
Ethernet 10 Gb/s Intel X520-DA2
4 GPU NVidia GTX 980 4GB GDDR5
PSU 2x1600 W

supergpu3,supergpu4
Supermicro 7048GR-TR
MB Supermicro X10DRG-Q
2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache
RAM DDR4-2133 64GB (4 channels)
SSD Crucial 250GB
Ethernet 1 Gb/s
4 GPU NVidia GTX 1080 8GB GDDR5
PSU 2x2000 W

superphi1,superphi2
Supermicro 7048GR-TR
MB Supermicro X10DRG-Q
2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache
RAM DDR4-2133 64GB (4 channels)
SSD Crucial 250GB
Ethernet 1 Gb/s
1 GPU NVidia GTX 980 4GB GDDR5
3 Intel Xeon Phi
PSU 2x2000 W

dellgpu1,dellgpu2
Dell T620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64GB (4 channels)
HDD WD RE4 500GB
LAN Broadcom 57810S 2 x 10Gb/s SFP+
4 GPU NVidia GTX 980 4GB GDDR5
PSU 2x1100 W

File servers


matylda1-6
Supermicro SC113TQ
MB Supermicro X10DRW-E
2 CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache
RAM DDR4-2133 64GB
HDD Intel SSD 80 GB
Ethernet 10 Gb/s Intel X520-DA2
72 TB RAID-6 3Ware (LSI) 9750SA-8e
External JBOD Supermicro SC836E16-R1200B

scratch1-6
Supermicro SC815TQ
MB Supermicro X9DRW-iF
CPU Intel Xeon E5-2630, 2,3 GHz, 6/12core, 15 MB cache
RAM DDR3-1600 16 GB (4 channels)
Ethernet 10 Gb/s Intel X520-DA2
10 TB RAID-6 3Ware (LSI) 9750SA-8e
External JBOD Supermicro SC826E16-R500LP

Network interconnect 10Gb/s


cluster switches
Extreme Networks Summit X650-24x
4 x 24 ports Ethernet 10 Gb/s SFP+
SummitStack 256 Gb/s ring


HPC switch2 traffic


HPC switch3 traffic

Your IPv4 address: 54.198.58.62
Switch to IPv6 connection

DNSSEC [dnssec]