In addition to its large supercomputers, IT4Innovations also operates smaller complementary systems. We launched the first of our complementary systems in the Autumn of 2022.
These systems represent emerging, non-traditional, and highly specialized hardware architectures that are not yet common in supercomputing data centres.
New programming models, libraries, and application development tools are also deployed in complementary systems to extract maximum performance from this hardware. Complementary systems thus allow research teams to test and compare experimental architectures with traditional architectures (e.g., x86 + Nvidia GPUs) and optimise and accelerate computations in new research areas.
Complementary systems consist of several parts. All of these parts are built on servers from Hewlett Packard Enterprise and were supplied and implemented by M Computers from Brno.
Petr Plodík, Sales Director at M Computer, adds: “The complementary systems project incorporates the latest and most discussed experimental HPC and AI platforms. Some of them represent the first-ever deliveries of these technologies to the Czech Republic. It was our pleasure to deliver and implement them at IT4Innovations".
Technical specification of complementary systems, which consist of several hardware platforms:
Compute partition 1 – ARM A64FX processors
The compute nodes of the first part of complementary systems are built on ARM A64FX processors with integrated fast HBM2 memory. It is a fragment of one of the world's most powerful supercomputers in recent years, Fugaku, installed at the RIKEN Center of Computational Science in Japan (currently the second most powerful supercomputer). The configuration consists of eight HPE Apollo 80 compute nodes interconnected by a 100Gb/s Infiniband network.
Configuration of each compute node:
- 1× ARM A64FX, 48 cores, 2 GHz, 32 GB HBM2 memory
- 400 GB SSD
- HDR Infiniband 100 Gb/s
Two compute nodes equipped with ARM64FX
Compute partition 2 – Intel processors, Intel PMEM
The compute nodes in this part of complementary systems are based on Intel technologies. The servers are equipped with third-generation Intel Xeon processors and persistent (non-volatile) Intel Optane memory with a total capacity of 2TB and 8TB per server.
This part consists of two HPE ProLiant DL380 Gen 10 Plus nodes in the configuration:
- 2× Intel Xeon Gold 6338, 32 cores, 2 GHz
- 256 GB RAM
- 8 TB and 2 TB Intel Optane Persistent Memory (NVDIMM)
- 3,2 TB NVMe SSD
- HDR Infiniband 100 Gb/s
Intel Optane Persistent Memory
Compute partition 3 – AMD processors, AMD accelerators, AMD FPGA (Xilinx)
The third part of complementary systems is built on AMD technologies. The servers are equipped with third-generation AMD EPYC processors, four AMD Instinct MI100 GPU cards interconnected by a fast bus (AMD Infinity Fabric), and two Xilinx Alveo FPGA cards, both being different in performance. Xilinx is one of AMD's latest significant acquisitions. This part consists of two HPE Apollo 6500 Gen 10+ nodes in the following configuration:
- 2× AMD EPYC 7513, 32 jader, 2,6 GHz
- 256 GB RAM
- 3,2 TB NVMe SSD
- 4× AMD Instinct MI100 (AMD Infinity Fabric Link)
- FPGA Xilinx Alveo U250
- FPGA Xilinx Alveo U280
- HDR Infiniband 100 Gb/s
CPU partition of the HPE Apollo 6500 Gen10 Plus server
Compute partition 4 – Edge server
Complementary systems also include the HPE EL1000 edge server, designed to process AI jobs directly at the data source, often outside the data centre. The server has high computing power for AI inference thanks to the NVIDIA Tesla T4 GPU accelerator, several technologies for communication (10Gb Ethernet, Wifi, LTE), and low power consumption.
- 1× Intel Xeon D-1587, 16 jader, TDP 65W
- 1x NVIDIA Tesla T4, 16 GB, TDP 70W
- 128 GB RAM
- 1,92 TB SSD storage
- Interconnect:
- 2x 10 Gbps Ethernet,
- WiFi 802.11ac,
- LTE connectivity
- Power consumption of up to 500W
HPE Edgeline EL1000
Network Infrastructure
The interconnection of individual nodes of complementary systems is provided by the high-speed, low-latency Infiniband HDR interconnection network, built on an Nvidia/Mellanox switch with forty ports and a speed of up to 200 Gb/s. The infrastructure also includes a 10Gb Ethernet network.
Software
An important part of complementary systems is software, including environments, compilers, numerical libraries, algorithm development, and debugging tools.
HPE Cray Programming Environment
The HPE Cray Programming Environment is a comprehensive tool for developing HPC applications in a heterogeneous environment. It supports all complementary systems architectures. It includes optimised libraries, support for the most widely used programming languages, and several tools for analysing, debugging, and optimising parallel algorithms.
Intel oneAPI
OneAPI is Intel's tool for developing applications deployed on heterogeneous platforms - CPU, GPU, and FPGA. It is planned to be used primarily for FPGA cards in complementary systems.
AMD ROCm
ROCm is an AMD software package that includes programming models, development tools, libraries, and integration tools for the most widely used AI frameworks that run on top of AMD GPU accelerators.