Computing cluster
HPC cluster is based on blade systems from IBM and Dell. Dell blade
servers were purchased within VAVINET
project, blade servers from IBM within projects of FR CESNET
and FRVŠ. Together there is now 102
modules, each with two 4 to 28 core CPUs and 8 to 256GB of RAM. On all
blade servers we use 64bit CentOS operating system. To store large datasets
there are 12 storage servers with total capacity of over 500TB.
All computations on the cluster are maintained by the Sun Grid Engine (SGE)
job management system. The tasks have to be described with a script (usually
a shell script) and this script is submitted to SGE. Based on current state
of computational nodes as well as storage and other conditions SGE decides
when and on which node to assign each task. Maximal theoretic size of a
parallel computation is dynamic, depends on the current state of all nodes
and currently it could be as high as 2400 processes.
Blade servers
Chassis | Modules | Hardware |
 blademm1 |
 blade001 - blade012 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
12 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels) - blade001-005,009,010,012
RAM DDR3 30GB (3 channels) - blade006-008,011
HDD SAS 2,5" 146 GB
|
 blademm2 |
 blade013 - blade024 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
12 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels)
HDD SAS 2,5" 146 GB - blade013-018
HDD SAS 2,5" 300 GB - blade019-024
|
 blademm3 |
 blade025 - blade029 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
5 modules HS22 (7870)
2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB
RAM DDR3 36GB (3 channels)
HDD SAS 2,5" 300 GB
|
 blade030 - blade036 |
7 modules HS21XM (7995)
2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2
RAM FB-DDR2/5300 20GB (2 channels)
HDD SAS 2,5" 300 GB
|
 blademm4 |
 blade037 - blade048 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
12 modules HS21XM (7995)
2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2
RAM FB-DDR2/5300 8 GB (2 channels)
HDD SCSI 73 GB
|
 dellmm1 |
 blade055 - blade070 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
6 modules Blade Dell M630
2 CPU Intel Xeon E5-2680 v3 2,4 GHz, 14/28core, 35 MB cache
RAM DDR4-2133 128 GB (4 channels)
HDD 2,5" 300 GB SAS 15k
LAN Broadcom 57810-k 2 x 10Gb/s KR
|
10 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR
|
 dellmm2 |
 blade071 - blade086 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
16 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR
|
 dellmm3 |
 blade087 - blade102 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
14 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64/128 GB (4 channels)
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR
|
2 modules Blade Dell M620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 256 GB
HDD 2,5" 300 GB SAS 10k
LAN Broadcom 57810-k 2 x 10Gb/s KR
|
HPC servers
 welda1, welda2, ulrika, svatava |
Supermicro SC113TQ
MB Supermicro X10DRW-E
CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache
RAM DDR4-2133 32GB (4 channels)
HDD Crucial SSD 250 GB
PSU 600W
|
GPU servers
 supergpu1,supergpu2 |
Supermicro 7047GR-TPRF
MB Supermicro X9DRG-QF
2 CPU Intel Xeon E5-2630v2 2,6 GHz, 6/12core, 15 MB cache
RAM DDR3-1600 64GB (4 channels)
HDD WD RE4 500GB
Ethernet 10 Gb/s Intel X520-DA2
4 GPU NVidia GTX 980 4GB GDDR5
PSU 2x1600 W
|
 supergpu3,supergpu4 |
Supermicro 7048GR-TR
MB Supermicro X10DRG-Q
2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache
RAM DDR4-2133 64GB (4 channels)
SSD Crucial 250GB
Ethernet 1 Gb/s
4 GPU NVidia GTX 1080 8GB GDDR5
PSU 2x2000 W
|
 superphi1,superphi2 |
Supermicro 7048GR-TR
MB Supermicro X10DRG-Q
2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache
RAM DDR4-2133 64GB (4 channels)
SSD Crucial 250GB
Ethernet 1 Gb/s
1 GPU NVidia GTX 980 4GB GDDR5
3 Intel Xeon Phi
PSU 2x2000 W
|
 dellgpu1,dellgpu2 |
Dell T620
2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache
RAM DDR3-1600 64GB (4 channels)
HDD WD RE4 500GB
LAN Broadcom 57810S 2 x 10Gb/s SFP+
4 GPU NVidia GTX 980 4GB GDDR5
PSU 2x1100 W
|
File servers
 matylda1-6 |
Supermicro SC113TQ
MB Supermicro X10DRW-E
2 CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache
RAM DDR4-2133 64GB
HDD Intel SSD 80 GB
Ethernet 10 Gb/s Intel X520-DA2
72 TB RAID-6 3Ware (LSI) 9750SA-8e
External JBOD Supermicro SC836E16-R1200B
|
 scratch1-6 |
Supermicro SC815TQ
MB Supermicro X9DRW-iF
CPU Intel Xeon E5-2630, 2,3 GHz, 6/12core, 15 MB cache
RAM DDR3-1600 16 GB (4 channels)
Ethernet 10 Gb/s Intel X520-DA2
10 TB RAID-6 3Ware (LSI) 9750SA-8e
External JBOD Supermicro SC826E16-R500LP
|
Network interconnect 10Gb/s
 cluster switches |
Extreme Networks Summit X650-24x
4 x 24 ports Ethernet 10 Gb/s SFP+
SummitStack 256 Gb/s ring
|
HPC switch2 traffic
HPC switch3 traffic
|