Skip to content

Storage Hardware Overview

The Perun HPC cluster is equipped with a multi-tiered storage architecture designed to balance performance, capacity, and scalability. This document provides an overview of the underlying storage hardware used across different tiers.

If you are looking for filesystem-level information (mount points, quotas, usage), refer to Available Mountpoints.

Home and Apps Storage Configuration

The Home (/home) and Apps (/apps) storage tiers are designed for user environments and centrally installed software. These systems prioritize reliability and consistency over raw performance.

1× DDN IntelliFlash N6200 (2U):

  • Dual active/active controllers (fully redundant)
  • 4× 100GbE ports (NFS export)
  • 1GbE management interfaces per controller
  • IntelliFlash OS with NFS (v3/v4)
  • Inline deduplication and compression
  • Snapshot and cloning support

DDN IntelliFlash N6200

Fig. 1. DDN IntelliFlash N6200 storage system

Drive configuration:

  • 24× NVMe SSD (13.97 TiB per drive)

Pool layout:

  • pool-1: 12 NVMe drives (~130 TiB usable)
  • pool-2: 12 NVMe drives (~130 TiB usable)

Capacity:

  • Raw: ~335 TiB (~368 TB)
  • Usable: ~260 TiB

Note

Both /home and /apps are exported via NFS and are not intended for high-throughput parallel workloads.

Scratch Storage Configuration

The Scratch (/scratch) storage tier is designed for high-performance temporary data access and provides the highest I/O throughput.

Warning

Scratch storage is not backed up and is subject to purge policies. Users must transfer important results after job completion.

4× DDN SFA400NVX2T (2U each):

  • 4× 3rd Gen Intel Xeon Scalable Processors per appliance
  • 2 controllers per appliance (Active/Active)
  • 2 VMs per controller (16 VMs total: scrp-vm01–scrp-vm16)
  • DeClustered RAID (DCR)
  • 2× InfiniBand NDR200 (200 Gb/s) per VM
  • Total: 32 InfiniBand ports
  • LNET: o2ib0
  • SFAOS 12.8, EXAScaler (Lustre)

DDN SFA400NVX2T with bezel

Fig. 2. DDN SFA400NVX2T (with bezel)

DDN SFA400NVX2T without bezel

Fig. 3. DDN SFA400NVX2T (without bezel)

Drive configuration:

  • 96× NVMe SSD (15.36 TB per drive)

Lustre layout:

  • 16× MDT, 32× OST

Capacity:

  • Raw: ~1.47 PB
  • Usable: ~1.01 PiB
  • Inodes: ~12.9 billion

Project Storage Configuration

The Project (/project) storage tier provides shared storage for research groups, combining NVMe performance with HDD capacity.

2× DDN SFA400NVX2 (2U each):

  • 4× 3rd Gen Intel Xeon Scalable Processors per appliance
  • 2 controllers per appliance (Active/Active)
  • 2 VMs per controller (8 VMs total: prjp-vm01–prjp-vm08)
  • DeClustered RAID (DCR)
  • 2× InfiniBand NDR200 (200 Gb/s) per VM
  • Total: 16 InfiniBand ports
  • SAS4 expansion connectivity
  • LNET: o2ib0
  • SFAOS 12.8, EXAScaler (Lustre)

DDN SFA400NVX2

Fig. 4. DDN SFA400NVX2 controller

8× DDN SS9024 Disk Enclosures (4U each):

  • 90 drive slots per enclosure
  • 2× SAS4 I/O modules
  • Redundant power and cooling

DDN SS9024

Fig. 5. DDN SS9024 expansion enclosure

Drive configuration:

  • 24× NVMe SSD (15.36 TB)
  • 360× SAS HDD (18 TB)

Lustre layout:

  • 8× MDT (NVMe)
  • 16× OST (HDD)
  • 16× OST (NVMe)

Capacity:

  • Raw HDD: ~6.48 PB
  • Raw NVMe: ~368 TB
  • Usable: ~9.33 PiB
  • Inodes: ~24+ billion

Work Storage Configuration

The Work (/work) storage consists of local NVMe SSDs attached to compute nodes. It provides very high I/O performance for temporary data.

Warning

Data in /work is not persistent and is removed after job completion or node reboot.

The configuration depends on node groups:

Node Range / Type Model Capacity per Node
cn001–cn045 Samsung MZ1L23T8HBLA-00A07 3.84 TB
cn046–cn060 Micron 7450 (MTFDKBZ7T6TFR) 7.68 TB
gn001-gn076 Micron 7450 (MTFDKBG3T8TFR) 3.84 TB

General characteristics:

  • Local (not shared across nodes)
  • No quotas
  • Not accessible across jobs

Note

Use /work for temporary high-I/O data only. Persistent data should be stored in /project or /home.

Notes on Architecture

Perun storage uses EXAScaler (Lustre) for parallel filesystems and NFS (IntelliFlash) for user environments.

  • Lustre (/scratch, /project) → large-scale parallel I/O
  • NFS (/home, /apps) → user environment and small files

Note

Lustre performs best with large, sequential I/O. NFS is not suitable for heavy parallel workloads.

Created by: Andrej Sec