In addition to its large supercomputers, IT4Innovations also operates smaller complementary systems.

These systems represent emerging, non-traditional, and highly specialized hardware architectures that are not yet common in supercomputing data centres.

New programming models, libraries, and application development tools are also deployed in complementary systems to extract maximum performance from this hardware.
These systems thus allow research teams to test
and compare experimental architectures with traditional architectures (e.g., x86 + Nvidia GPUs) and optimise
and accelerate computations in new research areas.

 

 

Technical specifications of complementary systems, which consist of several hardware platforms:

 

Compute partition 1 – Arm A64FX processors

The compute nodes of the first part of complementary systems are built on Arm A64FX processors with integrated fast HBM2 memory. It is a fragment of one of the world's most powerful supercomputers in recent years, Fugaku, installed at the RIKEN Center of Computational Science in Japan (currently the second most powerful supercomputer). The configuration consists of eight HPE Apollo 80 compute nodes interconnected by a 100Gb/s Infiniband network.

Configuration of each compute node:

  • 1× Arm A64FX, 48 cores, 2 GHz, 32 GB HBM2 memory
  • 400 GB SSD
  • HDR Infiniband 100 Gb/s

 

 

Two compute nodes equipped with ARM64FXDva výpočetní uzly s ARM64FX

 

 

Compute partition 2 – Intel processors, Intel PMEM, Intel FPGA (Altera)

The compute nodes in this part of complementary systems are based on Intel technologies. The servers are equipped with third-generation Intel Xeon processors and persistent (non-volatile) Intel Optane memory with a total capacity of 2TB and 8TB per server.

This part consists of two HPE ProLiant DL380 Gen 10 Plus nodes in the configuration:

  • 2× Intel Xeon Gold 6338, 32 cores, 2 GHz
  • 256 GB RAM
  • 8 TB and 2 TB Intel Optane Persistent Memory (NVDIMM)
  • 3,2 TB NVMe SSD
  • HDR Infiniband 100 Gb/s

 

Intel Optane Persistent Memory

 

Compute partition 3 – AMD processors, AMD accelerators, AMD FPGA (Xilinx)

The third part of complementary systems is built on AMD technologies. The servers are equipped with third-generation AMD EPYC processors, four AMD Instinct MI100 GPU cards interconnected by a fast bus (AMD Infinity Fabric), and two Xilinx Alveo FPGA cards, both being different in performance. Xilinx is one of AMD's latest significant acquisitions. This part consists of two HPE Apollo 6500 Gen 10+ nodes in the following configuration:

  • 2× AMD EPYC 7513, 32 jader, 2,6 GHz
  • 256 GB RAM
  • 3,2 TB NVMe SSD
  • 4× AMD Instinct MI100 (AMD Infinity Fabric Link)
  • FPGA Xilinx Alveo U250
  • FPGA Xilinx Alveo U280
  • HDR Infiniband 100 Gb/s

 

CPU partition of the HPE Apollo 6500 Gen10 Plus server

 

Compute partition 4 – Edge server

Complementary systems also include the HPE EL1000 edge server, designed to process AI jobs directly at the data source, often outside the data centre. The server has high computing power for AI inference thanks to the NVIDIA Tesla T4 GPU accelerator, several technologies for communication (10Gb Ethernet, Wifi, LTE), and low power consumption.

  • 1× Intel Xeon D-1587, 16 jader, TDP 65W
  • 1x NVIDIA Tesla T4, 16 GB, TDP 70W
  • 128 GB RAM
  • 1,92 TB SSD storage
  • Interconnect:
    • 2x 10 Gbps Ethernet, 
    • WiFi 802.11ac,
    • LTE connectivity
  • Power consumption of up to 500W

 

HPE Edgeline EL1000

Network Infrastructure

The interconnection of individual nodes of complementary systems is provided by the high-speed, low-latency Infiniband HDR interconnection network, built on
an Nvidia/Mellanox switch with forty ports and a speed of up to 200 Gb/s. The infrastructure also includes a 10Gb Ethernet network.

Software

An important part of complementary systems is software, which includes environments, compilers, numerical libraries, and algorithm development and debugging tools.

HPE Cray Programming Environment

The HPE Cray Programming Environment is a comprehensive tool for developing HPC applications in a heterogeneous environment. It supports all complementary systems architectures. It includes optimised libraries, support for the most widely used programming languages, and several tools for analysing, debugging, and optimising parallel algorithms.

Intel oneAPI

OneAPI is Intel's tool for developing applications deployed on heterogeneous platforms - CPU, GPU, and FPGA. It is planned to be used primarily for FPGA cards in complementary systems.

AMD ROCm

ROCm is an AMD software package that includes programming models, development tools, libraries, and integration tools for the most widely used AI frameworks that run on top of AMD GPU accelerators.