Modern GPU servers already contain substantial high-performance NVMe capacity. At the same time separate high-performance storage appliance is usually used in GPU servers cluster as Tier-0 data storage. The opportunity is to make local NVMe capacity useful to the entire cluster.

Xinnor demonstrated at the Weizmann Institute of Science an architecture that turns NVMe installed in GPU/HPC nodes into shared, protected storage without adding a separate high-performance storage appliance.

The architecture needed to provide:

• A single global namespace to applications the cluster wide.
• A data path suitable for both traditional HPC and GPU-intensive workloads.
• High aggregate throughput and IOPS.
• Protection against drive, server node, and network failures.
• Predictable performance during degraded operation.

The resulting architecture turns storage embedded in the compute infrastructure into a shared Tier-0 resource, while keeping compute and storage physically converged.

xiRAID Opus provides the underlying block storage layer for this architecture that includes:

• User-space NVMe driver and NVMe-RDMA/TCP backend fabric layer (target and initiator) to export local NVMe drives and share them between server nodes as backend block devices.
• RAID protection module to combine shared block devices to fault-tolerant and scale-out block storage platform.
• NVMe-RDMA/TCP frontend fabric with NVMe multipath to expose cluster wide block namespaces to file system layer or applications.
• High-availability and management to automatic storage and frontend namespaces failover and failback in case of a server node fail or restart as well as to automate RAID data reconstruction.

At the Weizmann Institute of Science, IBM Storage Scale was already the established file-system environment. On top of xiRAID Opus, IBM Storage Scale is deployed as the clustered file system layer, providing a unified storage namespace and shared file access for the HPC/GPU compute infrastructure.

• 5 GPU/HPC nodes
• 4 × Micron 7450 NVMe SSDs per node
• 8 × NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs per node
• NVIDIA ConnectX-7 networking
• Dual 400 Gb/s InfiniBand connectivity
• xiRAID Opus as the distributed RAID and NVMe-oF storage layer
• 5 distributed RAID 5 (3+1) arrays
• Separate backend and frontend NVMe-oF/RDMA storage paths
• IBM Storage Scale 6.0.1
• 5 NSDs accessed directly from all participating nodes
• 14.3 TiB shared file system with a 1 MiB file-system block size
• NVIDIA GPUDirect Storage for direct storage-to-GPU data transfers

The configuration details are shown at the figure 1 and includes:

1. Five servers (A, B, C, D, E) with 4 local NVMe drives in each server. At the figure each drive is named as a server letter and a number. For example, server A has drives A1, A2, A3 and A4.
2. xiRAID Opus detaches drives from Linux drivers and exports the local NVMe drives or their partitions to the backend NVMe-oF fabric to make the drives visible at each node.
3. xiRAID Opus initiator imports required block devices from the backend NVMe-oF fabric to use them as underlined block devices for a RAID protection layer. For example, green drives B1, C1, D1, E1 at the server A are imported from servers B, C, D, E respectively.
4. RAID layer combines imported block devices to RAID 5 or 6 to enable data protection and continue IO in case of a remote server fail. In the RAID 5 example shown, each volume is composed of one drive from each of the other four servers, distributing both data and failure domains across the cluster.
5. Frontend clustered NVMe target represents each RAID as a cluster visible namespace with automated multipath. The namespace can be accessed by an application initiator at any server using frontend NVMe fabric. An NVMe namespace for each RAID is configured at each server but one namespace only (green colour) is active while pathes to others (brown colour) are reported as inaccessible by multipath infrastructure. This allows seamless switching application IO to the namespace backing device in case of a server fail and the RAID failover.
6. To make the RAID volumes available to applications running on the same servers, the in-kernel NVMe-oF multipath initiator is used. If an application provides its own clustering and high-availability mechanisms, no additional application-level resource manager is required from Opus. IBM Storage Scale (GPFS) is the application used in the validated architecture. It connects all RAIDs, provided by Opus layer, at each server by frontend fabric. Storage Scale cluster software manages application-level clustering and workload placement.

Figure 1. Five-node architecture: node-local NVMe is aggregated by xiRAID Opus and
presented to IBM Storage Scale as shared storage

The figure 2 represents server B fail and corresponded data storage and IO paths failover:

1. All the local drives of Server B become unavailable, so RAIDs A, C, D, and E, become degraded.
2. The active path to RAID B disappears. The remaining paths are present, but their state are still inaccessable. As a result, I/O to RAID B is reported as busy while no usable path is available.
3. The xiRAID Opus cluster detects the failure of Server B and initiates failover. RAID B is reactivated on one of the surviving servers. In this example, it is reactivated on Server A.
4. After RAID B becomes active on Server A, the corresponding namespace path changes from inaccessible to active. Linux native NVMe multipathing resumes I/O to RAID B through Server A. From the application perspective, the outstanding requests complete after a temporary increase in latency.

Figure 2. Node-failure scenario: RAID ownership moves to a surviving node while the
remaining arrays continue in degraded mode.

Therefore, xiRAID Opus detaches locally attached NVMe resources from individual nodes and composes them into protected, distributed storage pools using software-defined RAID. Logical volumes provisioned from these pools are exposed across the cluster as shared NVMe namespaces.

IBM Storage Scale is used as the shared file-system layer because it is the established file-system environment at the Weizmann Institute. xiRAID Opus presents shared block resources below the file-system layer, allowing this architecture to integrate with enterprise clustered file systems appropriate to a given environment.

Storage Scale combines the xiRAID Opus volumes into a shared parallel file system with a single global namespace. In the Weizmann Institute configuration, participating nodes access the NSDs directly over NVMe-oF/RDMA, avoiding an additional storage-server hop in the steady-state data path.

IBM Storage Scale preserved essentially all of the sequential bandwidth available from the underlying distributed NVMe-oF layer.

The storage was measured progressively, first at the raw distributed block layer and then through the complete shared file system.

For bandwidth-intensive workloads, the most important comparison is between the two sequential results. Storage Scale delivered 131.7 GiB/s read versus 131.2 GiB/s at the raw block layer, while sequential write retained approximately 98.5% of raw NVMe-oF bandwidth.

For checkpointing, dataset streaming, and other large-block HPC and AI workloads, applications therefore gain shared access and a global namespace without introducing a meaningful sequential-bandwidth penalty.

The file system also reached 5.83 million 4 KiB random-read IOPS and 3.12 million random-write IOPS, extending the architecture beyond purely bandwidth-oriented workloads.

The architecture remained available after the loss of an entire node, with performance degradation following the loss of physical resources in a predictable way.

Losing one node in this converged design affects both compute and storage resources. During the controlled failure test, the affected RAID resource was moved to a surviving node while the other arrays continued operating in degraded mode.

Random-I/O performance fell almost exactly in proportion to the reduction from five nodes to four, while average random-read and random-write latency increased by only 0.6%. Sequential workloads retained approximately 70% of healthy bandwidth, reflecting the additional RAID work required in degraded mode.

The shared file system remained mounted and available during the controlled failure test, with no observed data loss. Once the failed resources were restored, RAID reconstruction returned the storage to full redundancy within minutes.

The results demonstrate predictable performance in degraded-mode rather than treating node failure as an unknown performance event.

The same architecture delivered up to 132.1 GiB/s from IBM Storage Scale into GPU memory using NVIDIA GPUDirect Storage.

Storage-to-GPU path required by modern AI workloads was run using xiRAID Opus. NVIDIA GPUDirect Storage enables data to reach GPU memory through DMA without first being staged through a CPU memory buffer.

Importantly, direct DMA was verified rather than inferred from throughput. In a single-GPU test, all 124,012 measured operations used the NVIDIA nvfs path with zero POSIX fallbacks. GPU_DIRECT achieved 22.1 GiB/s, compared with 18.3 GiB/s through the CPU_ONLY path — a 21% improvement.

Across five nodes using one GPU per node, aggregate GPU_DIRECT read bandwidth reached 132.1 GiB/s.

The result shows that disaggregating NVMe does not prevent the storage architecture from supporting GPU-centric data paths. The shared file system can feed GPU memory at approximately the same aggregate rate as its maximum measured sequential read throughput.

The project established a practical reference architecture for turning NVMe inside GPU/HPC nodes into shared Tier-0 storage, with measured performance, availability, and GPU-access characteristics.

The main outcomes go beyond headline benchmark numbers:

1. Existing NVMe capacity becomes a shared resource. Storage embedded in compute nodes can be used across the cluster through a single global namespace instead of remaining isolated locally.

2. Shared storage does not have to mean a major bandwidth trade-off. In this configuration, IBM Storage Scale retained essentially all measured sequential NVMe-oF bandwidth.

3. Failure behavior is predictable. A complete node failure reduced available performance rather than causing loss of the shared file system.

4. Data protection and recovery are handled within the software-defined storage layer. RAID protection spans node failure domains, while restored resources can be rebuilt without redesigning or reprovisioning the file system.

5. The same architecture extends from traditional HPC to GPU data paths. GPUDirect Storage demonstrated that shared Tier-0 capacity can also feed accelerator memory at very high bandwidth.

6. The approach avoids treating storage and compute as two completely separate infrastructures. Compute nodes can contribute capacity to a common storage pool while continuing to serve application workloads.

The results of the validation demonstrate how the architecture performs across the three key requirements of a Tier-0 storage system: performance, resilience, and direct access to GPU. The following measurements show that the shared storage layer can maintain near-raw NVMe bandwidth, continue operating through a complete node failure, and deliver high-bandwidth data directly to GPUs.

With xiRAID Opus providing the distributed block layer beneath IBM Storage Scale, the system reached 131.7 GiB/s sequential read throughput, effectively matching the same performance measured at the raw NVMe-oF layer. Sequential write reached 46.9 GiB/s, retaining approximately 98.5% of the raw block-level bandwidth. For random 4KiB IO impressive 5.8M IOPS were achieved for read and 3.1M IOPS for write.

The same architecture was then tested through the controlled loss of an entire node. With four of five nodes remaining, the architecture retained approximately 70–80% of healthy-cluster sequential and random-I/O performance while average random-I/O latency increased by only 0.6% and average sequential latency increased by 10-15%. Therefore, validated solution can continue serving data with predictable degradation rather than turning a node loss into a storage outage.

Finally, the same shared Tier-0 architecture to GPU data access path was validated. Using NVIDIA GPUDirect Storage, aggregate reads from the shared file system into GPU memory reached 132.1 GiB/s across five nodes — effectively the same bandwidth as the maximum sequential read result of the shared storage system.

xiRAID Opus validation at Weizmann Institute of Science provides a concrete engineering reference for applying the same approach to other HPC and AI environments. The exact topology can be adapted to different cluster sizes, file systems, availability requirements, and workload profiles while preserving the core principle: use NVMe already present in compute infrastructure as high-performance shared storage.

Read the technical deep dive for the complete architecture, methodology, and benchmark results:

Read the blog post