Disaggregated Tier-0 Storage for HPC and AI: xiRAID Opus and IBM Storage Scale PoC

1. Background and PoC Objectives

The challenge is to create a high-performance, distributed, and fault-tolerant storage system that leverages commodity NVMe drives deployed directly in compute/GPU nodes, transforming local NVMe resources into a shared, scale-out storage platform for HPC and AI workloads.

Using these drives only as isolated local storage significantly reduces infrastructure flexibility. HPC and AI workloads vary widely in their capacity, performance, and data-retention requirements, while local-only storage binds capacity to individual nodes. This can leave usable storage stranded, create imbalances between nodes, or require additional data copying and movement. It is particularly inefficient for checkpointing: if checkpoints exist only on the node that created them, the loss of that node may also remove the most recent recovery points, forcing the workload to restart from an older checkpoint or, in the worst case, from the beginning.

The required solution must therefore aggregate node-local NVMe drives into a shared resource pool and allow storage capacity and performance to be flexibly redistributed across the cluster. It should preserve the economic advantages of commodity local NVMe while delivering the performance, availability, and operational flexibility expected from a dedicated HPC storage platform—without a dedicated external storage appliance.

The solution is expected to provide:

• A single global namespace across all compute and GPU nodes
• Node, drive, and network fault tolerance
• No single point of failure
• Scale-out architecture
• Near-local-NVMe performance
• High aggregate IOPS and throughput
• Better economics than dedicated storage appliances

1.1. Original Concept

The original concept was to transform the NVMe drives already installed in GPU/HPC servers into a shared, high-performance storage platform, eliminating the need for a traditional dedicated storage appliance. The goal was to reduce storage costs and hardware overhead while increasing available storage capacity and providing the performance required by demanding HPC and AI workloads.

Instead of using the NVMe drives as isolated local storage, the concept was to aggregate NVMe resources across multiple GPU/HPC nodes over a high-speed network, creating a distributed, fault-tolerant, and scale-out storage platform.

xiRAID Opus provides the underlying storage layer for this architecture. As a Linux user-space software solution, xiRAID Opus unifies local and network-attached NVMe drives into a high-performance storage platform, enabling NVMe resources from multiple nodes to be aggregated and presented as shared storage. This approach provides high performance, scalability, and fault tolerance while minimizing dedicated hardware, power consumption, and infrastructure complexity.

On top of xiRAID Opus, IBM Storage Scale was deployed as the clustered file system layer, providing a unified storage namespace and shared file access for the HPC/GPU compute infrastructure.

The resulting architecture combines local NVMe resources, high-speed networking, xiRAID Opus, and IBM Storage Scale into a software-defined, distributed storage platform. It transforms storage resources already available within GPU/HPC servers into a shared scale-out storage system, combining near-local-NVMe performance, high aggregate IOPS and throughput, fault tolerance, scalability, and improved economics compared with traditional dedicated storage appliances.

This document describes a proof-of-concept (PoC) for creating distributed Tier-0 storage based on the xiRAID Opus solution using the Weizmann Institute infrastructure and GPU servers. It includes the performance measurements of the proposed solution, a simple technical description of the architecture, and a discussion of the limitations of the current implementation.

2. Architecture Overview

xiRAID Opus is a Linux user-space software solution that unifies local and network-attached NVMe drives into a high-performance, storage platform with high-performance RAID technology for drive-failure protection and an integrated NVMe-oF subsystem. It maximizes speed and reliability for demanding applications while minimizing hardware overhead, lowering power costs, and simplifying infrastructure management.

Disaggregated xiRAID Opus uses a layered, disaggregated architecture in which locally attached NVMe drives are shared across a cluster, combined into RAID volumes on peer nodes, and then re-exported to all cluster nodes as shared NVMe namespaces. This separates physical drive placement, RAID ownership, and application placement. The following sections walk through the data path from the physical NVMe drives up to the application layer and then illustrate a server failover scenario. As an example, consider a simple configuration with 5 servers (A, B, C, D, E) with 4 local NVMe drives in each server (figure 1).

Figure 1

Each drive has a unique name in our example: a server letter and a number. For example, Server A has drives A1, A2, A3 and A4. Each server runs a xiRAID Opus instance. These instances are combined into a cluster which enables RAID/volume failover and failback, as well as ANA multipath support (see below).

2.1. Backend drives sharing

At the infrastructure layer, xiRAID Opus transforms node-local NVMe drives into a cluster-wide, disaggregated storage resource pool. On each server, an xiRAID Opus instance imports the locally attached drives and exposes them over the backend NVMe-oF fabric, making every device accessible to the Opus engine at all servers in the cluster (Figure 2). This eliminates the boundaries of individual nodes and provides the shared storage foundation for the upper layers of the platform. The imported drives are disconnected from local Linux drivers and cannot be accessed by applications outside xiRAID Opus, helping prevent accidental data corruption.

Figure 2

2.2. Backend drives import and RAID construction

At the second stage, xiRAID Opus composes the disaggregated NVMe resources into networked, fault-tolerant RAID devices. Each xiRAID Opus instance imports the required remote NVMe drives from the cluster-wide resource pool and compose them into a RAID volume owned and managed by that server (Figure 3). In the RAID 5 example shown, each volume is composed of one drive from each of the other four servers, distributing both data and failure domains across the cluster.

Figure 3

The following placement rules are used when constructing RAID volumes:

1. A RAID owned by Server N must not contain drives physically attached to Server N.

   This provides a uniform backend access path for all RAID members and, more importantly, avoids coupling the loss of the RAID owner with the loss of one of its member drives. If Server N fails, only the RAID engine must be failed over; none of that RAID's member drives are lost as a direct result of the same server failure. This rule helps to avoid unrecoverable RAID stripe failures caused by write-hole scenarios.

2. A RAID must not contain more drives from any single server than its RAID level can tolerate losing.

   Because a server failure makes all NVMe drives attached to that server unavailable at the same time, each server represents a failure domain. Therefore, a RAID 5 volume may contain at most one drive from any server, while a RAID 6 volume may contain at most two.

2.3. RAID placement rules

At this layer, xiRAID Opus creates the RAID volumes on each serverusing the imported drives and the principles described above (figure 4).

Figure 4

2.4. Clustered NVMe-oF target

At the frontend layer, the RAIDs are exported over the frontend network through the Opus clustered NVMe-oF target. Each cluster node provides a path to every RAID namespace. For a RAID currently owned by a particular node, the path through that node is reported in the ANA optimized state, while paths through the other nodes are reported as inaccessible (Figure 5).

Figure 5

When RAID ownership moves from one node to another, for example during failover, failback, or planned relocation, the corresponding ANA states are updated accordingly.

2.5. NVMe-oF Initiator and multipathing

To make the RAID volumes available to applications running on the same servers, the in-kernel NVMe-oF initiator is used. Linux native NVMe multipathing presents each RAID namespace as a single block device with multiple underlying NVMe-oF paths (figure 6).

Figure 6

As a result, all the servers have access to all the RAIDs (the RAIDs are available on these servers as standard block devices). Each RAID volume behaves as a SAN-shared LUN/volume/namespace accessible from all servers. So, at this point the systems are ready for application deployment.

2.6. IBM Storage Scale integration

If an application provides its own clustering and high-availability mechanisms, no additional application-level resource manager is required from Opus. IBM Storage Scale (GPFS) is one example. Its own cluster software manages application-level clustering and workload placement. The only requirement is a set of block devices shared among all servers, which are provided by all the previous layers. This configuration is shown in Figure 7.

Figure 7

2.7. Failover

Let’s consider a failover process. The initial state of the cluster presented on Figure 8. In case of Server B failure, the following things will happen:

1. All the local drives of Server B will become unavailable, so RAIDs A, C, D, and E, each of which includes a drive exported from Server B, become degraded.
2. The optimized path to RAID B disappears. The remaining paths are still present, but they are in the ANA inaccessible state. As a result, I/O to RAID B is temporarily stalled while no usable path is available.
3. The xiRAID Opus cluster detects the failure of Server B and initiates failover.
4. RAID B is reactivated on one of the surviving servers. In this example, it is reactivated on Server A (Figure 8).
5. After RAID B becomes active on Server A, the ANA state of the path through Server A changes from inaccessible to optimized.

Linux native NVMe multipathing detects the ANA state change and resumes I/O to RAID B through Server A. From the application perspective, the outstanding requests complete after a temporary increase in latency.

Figure 8

3. Weizmann Institute PoC Deployment

Xinnor and the Weizmann Institute of Science deployed a 5-node disaggregated cluster running xiRAID Opus and IBM Storage Scale over NVMe-oF.

3.1. PoC goals:

• Demonstrate the feasibility of building a high-performance, Tier-0 distributed file system from NVMe drives installed locally in existing compute and GPU nodes, without dedicated RAID controllers or external storage arrays.
• Provide a single global namespace that makes the aggregated storage capacity accessible from every compute and GPU node in the cluster.
• Validate that the combined xiRAID Opus and IBM Storage Scale architecture delivers performance close to the underlying raw NVMe-oF block devices.
• Confirm that the file system remains available during a node failure, preserves access to data, and supports controlled failover, degraded operation, reconstruction, and failback.
• Quantify the performance impact of operating in healthy and degraded states to establish practical sizing and availability expectations for production HPC and AI workloads.

3.2. Environment

Clients (GPU/HPC compute nodes, Storage Scale GUI) connect over the front-end NVMe-oF/RDMA fabric to the IBM Storage Scale layer, which addresses 5 NSDs.

IBM Storage Scale supports two equally valid ways of reaching an NSD, and both were tested in this configuration; neither is a fallback for the other — either can be chosen per deployment:

• Direct (SAN) access — the client node sees the NSD itself over the storage fabric and issues block I/O to it directly, with no intermediate node in the data path.
• NSD server access — the client node sends the I/O request over the network to a designated NSD server, which performs the physical block I/O on its behalf and returns the data.

Both modes can also coexist in one file system: a node that has a direct path uses it, and a node that does not is served over the network. In the layout described here no dedicated NSD servers are deployed, so clients use direct access.

Each NSD is an NVMe-oF namespace exposing one RAID5 array in multipath ANA mode, where only the owner node's path is active/optimized. The storage layout is disaggregated: 5 RAID5 arrays (xnraid1…xnraid5) are built over a separate back-end fabric from partitions contributed by all 5 nodes, so every array has a member on every node.

3.3. Storage access model

In this design the storage nodes and the Storage Scale clients are not separate populations — they are the same 5 nodes, converged into a single Storage Scale layout in which every node both contributes partitions to the RAID5 arrays and mounts the file system. That collapses the usual trade-off: we keep every advantage of direct access while the drawbacks that normally argue for NSD servers do not apply.

• No extra hop, no shared bottleneck. Each node reaches the NSDs itself over the front-end NVMe-oF/RDMA fabric, so I/O takes the shortest path and aggregate bandwidth scales with the fabric rather than being funnelled through a few NSD servers.
• Fabric attachment costs nothing extra. Every node is already on the storage fabric by virtue of contributing its own partitions, so the per-client HBA/NIC, port and zoning cost that normally makes direct access expensive is already paid for.
• Client count is fixed and small. The scaling concern behind NSD servers — a growing client population each needing its own mapping — does not arise: the set of clients is closed at 5 and identical to the set of storage nodes.
• Configuration effort is bounded. Mapping, masking, multipath and device naming are done once across 5 symmetric nodes rather than being replicated across an open-ended client fleet.
• Failure domain is narrower. A lost path affects only the node that lost it, instead of taking down every client behind a failed NSD server; ANA multipath already handles path-level failover.
• No CPU tax on a shared server. Since there is no NSD server acting on behalf of others, no node carries someone else's I/O load on top of its own compute work.
• For these reasons direct (SAN) access is the primary mode for this deployment. NSD server access remains fully supported and was verified as well, and is the appropriate choice if clients outside this converged set — without access to the storage fabric — need to mount the file system later.

Tested array geometries included 3+1 (128k strip). The shared file system “xnrfs” provides ~14.3 TiB usable capacity with a 1 MiB GPFS block size, mounted read-write on all 5 nodes at /gpfs/xnrfs; initializing all 5 arrays in parallel takes about 2 minutes.

3.4. Deployment architecture

IBM Storage Scale is used over a SAN-attached NVMe-oF fabric. The cluster is built on a single deliberate idea: there is no separation between storage servers and compute clients. All five nodes are identical, all five contribute drives to the storage pool, all five own an array, all five export over the fabric, all five mount the file system, and all five run GPU workloads. Nothing in the design is a dedicated appliance. Redundancy lives in the RAID layer rather than in the file system, and every node reaches every NSD directly over the fabric rather than through an intermediary. The consequence is that a "storage node" and a "client node" are the same machine, and the architecture has to be read as a set of roles layered onto one hardware population rather than as tiers.

3.5. Node roles and networks

Each of node01–node05 simultaneously carries seven roles. This is the whole architecture in one table, and reading down the column is the fastest way to understand what the cluster is:

Networks. Three physically and functionally separate networks, which matters because it keeps control traffic off the data path:

Separating the GPFS daemon network from the data fabric means token traffic and metadata chatter cannot contend with bulk I/O for the same InfiniBand queues, and a saturated data path does not slow down cluster arbitration.

3.6. Storage geometry

Each array is RAID5 3+1: three data members plus one parity, 128 KB chunk, giving a 384 KiB full stripe and roughly 2,929 GiB usable per array — about 15 TB across the cluster. What makes the layout disaggregated is where the members come from. Array xnraid1 is built from one drive on node 2, one on node 3, one on node 4 and one on node 5; xnraid2from nodes 1, 3, 4, 5; and so on around the rotation. Every node therefore contributes exactly one member to four of the five arrays, plus owning one array outright.

This buys balanced load — no node is a hotspot, because every array's traffic is spread across four nodes' drives — but it also means no array is independent of any node. That single fact drives everything about the failure model below.

3.7. IBM Storage Scale configuration

The file system was created with all five NSDs in a single stanza:

%nsd:
  device=/dev/nvme0nX
  nsd=xnr_nsidNN
  usage=dataAndMetadata
  failureGroup=1
  pool=system
mmcrnsd -F /root/xnr_shared_nvme.nsd -v yes
mmcrfs xnrfs -F /root/xnr_shared_nvme.nsd -T /gpfs/xnrfs -A yes -B 1M -n 5 -m 1 -r 1 -M 1 -R 1
mmmount xnrfs -a

Four choices in that command define the storage policy, and each was made deliberately:

usage=dataAndMetadata, single system pool. Every NSD carries both data and metadata. With five identical, equally fast namespaces there is no reason to segregate metadata onto a faster tier — there is no faster tier. Metadata I/O spreads across all five arrays exactly as data does.

-B 1M. The file-system block is 1 MiB, chosen as a deliberate 2.67× of the 384 KiB stripe rather than an exact multiple. The alternative — rebuilding at 4+1 for a 512 KiB stripe and a clean 2.0× ratio — was built and benchmarked, and it lost 13% of sequential read and 35% of sequential write. Stripe alignment turned out not to be the thing that matters; the wider array spanning all five nodes' drives added cross-node RDMA fan-out that outweighed the parity-efficiency gain. A 512 KiB file-system block was also tested and rejected. 1 MiB stands.

-m 1 -r 1 -M 1 -R 1. No GPFS-level replication of either data or metadata, and the maximum replica count is fixed at 1 so it cannot be raised later without a rebuild. All redundancy is provided by RAID5 underneath.

failureGroup=1 for all five NSDs. Every NSD is declared to be in the same failure group. This is the point that deserves the most explanation, and it is not an oversight.

3.8. Failure domains

3.9. Verification

Each layer was verified independently rather than inferring health from the layer above. The checks below were run at deployment and are codified in the operational runbook.

Storage layer (xiRAID). xnr_cli raid show on each owner node, confirming each xnraidN reports online on its owner and configured elsewhere, with no degraded or absent members.

Transport layer (NVMe-oF). nvme list-subsys on every node, confirming all five targets reachable and all paths reporting live.

NSD layer. mmcrnsd -F ... -v yes at creation, which refuses to overwrite disks already carrying a descriptor — the guard against re-provisioning over live data. Then mmlsdisk xnrfs, confirming all five disks ready / up.

Access-path verification — the check that catches silent degradation. mmlsdisk xnrfs -M reports IO performed on node as a local device, and mmdiag --iohist shows lcl nvmeXnY entries rather than cli → 100.100.101.<server>.

Cluster layer. mmgetstate -a — all five daemons active; mmlsmount xnrfs -L — mounted on all five nodes, verified by IP; mmhealth node show — FILESYSTEM HEALTHY.

Availability. A full node failure was simulated end to end: node stopped, array migrated to a new owner with two commands, all five volumes verified still served, the surviving cluster benchmarked, and the node failed back with arrays auto-reconstructing. Zero data loss, no outage.

GDS. gdscheck -p on all five nodes, and — because a GDS call that falls back to the CPU path still reports success — true DMA was proven by counters (nvfs=124012, posix=0) and by an A/B in which GPU_DIRECT was measurably faster than CPU_ONLY, which a silent fallback could not be.

Summary

The architecture is five identical nodes, each simultaneously a drive contributor, an array owner, an NVMe-oF target and initiator, a GPFS quorum-manager, a client, and a GPU host. Storage is five declustered RAID5 3+1 arrays exported as five NVMe-oF namespaces and consumed as five NSDs in a single pool and a single failure group, with no file-system replication — all redundancy is RAID-level. Every node reaches every NSD directly, so there is no NSD server in the data path; converged mode adds server declarations that serve as free failover metadata without entering the steady-state path.

The design is validated at every layer independently, and its one hard limit is stated explicitly rather than left implicit: it tolerates a single node failure with roughly 70–80% of performance retained and zero data loss, and it has no margin for a second failure until that node returns.

4. Performance Benchmarking Methodology

The performance evaluation was organized as a single, progressive benchmark sequence. Each stage introduced an additional layer of the final architecture: first, the performance and fault behavior of an individual xiRAID array were measured; next, the complete distributed block layer was tested in raw NVMe-oF SAN mode; IBM Storage Scale was then added and evaluated in direct SAN access mode; and finally, the complete file system was benchmarked during a simulated node failure. This sequence made it possible to distinguish the performance characteristics of the RAID layer, the distributed block transport, the file system, and the high-availability mechanisms.

Benchmark sequence

Stage 1 — single-array raw block baseline.

The first stage isolated one raw xiRAID RAID5 3+1 volume and exercised it from a single node, before introducing either the cluster-wide SAN path or the file system. Fio was used with the libaio I/O engine, 16 jobs, and an I/O depth of 64 per job.

Sequential read and write tests used a 384 KiB block size, corresponding exactly to one xiRAID full data stripe of three 128 KiB data chunks. Random read and write tests used a 4 KiB block size. Each workload was executed in three array states:

1. healthy, with all RAID members available;
2. degraded, with one member missing;
3. reconstruction, while the missing member was being rebuilt under benchmark load.

This stage established the performance of an individual array, quantified the degradation caused by a missing RAID member, and measured the additional contention introduced by live reconstruction. Random 4 KiB write was included as the most demanding parity workload because small writes require RAID5 parity processing and are most sensitive to simultaneous reconstruction activity.

Stage 2 — cluster-wide raw NVMe-oF SAN.

The second stage evaluated the complete distributed block layer without a file system. All five NVMe-oF namespaces were accessed directly and driven in parallel from all five nodes. Tests were launched from node01 and distributed over SSH so that every participating node started the same workload concurrently.

Four workload classes were evaluated: 1 MiB sequential read, 1 MiB sequential write, 4 KiB random read, and 4 KiB random write. The 1 MiB workloads represented checkpoint, dataset streaming, and other large-block data transfers. The 4 KiB workloads represented small-block, metadata-intensive, and latency-sensitive access patterns.

The fio harness used --ioengine=libaio and --direct=1. All five namespaces were included in the device list using --filename=/dev/nvmeXn1:…:n5; the exact controller number differed on nodes where the local boot device affected NVMe enumeration. Process count was varied from 8 to 64 jobs per node, using workload-specific points including 8, 16, 24, 28, 48, and 64 jobs. I/O depth was varied alongside process count, with the total per-node queue depth calculated as:

Sequential tests used --offset_increment=40G to distribute jobs across different LBA ranges. Random tests used --norandommap to avoid maintaining an impractically large random block map. Read-only runs included --readonly as a safety guard.

Each measurement used --time_based --runtime=200 --ramp_time=4, producing a 200-second steady-state result after a four-second ramp period. Results were written in JSON format using --group_reporting --output-format=json, with one machine-readable result per node.

Stage 3 — IBM Storage Scale in direct SAN access mode.

The third stage repeated the performance evaluation through the xnrfs file system. All five nodes acted as Storage Scale clients and accessed the five NSDs directly over the NVMe-oF/RDMA fabric.

The following tuned Storage Scale configuration remained in force throughout this stage:

• pagepool 64G
• maxMBpS 40000
• workerThreads 1024
• nsdMaxWorkerThreads 1024
• aioWorkerThreads 2048
• nsdMultiQueue 512
• ignorePrefetchLUNCount yes
• numaMemoryInterleave yes
• file system block size: 1 MiB

Fio remained the primary saturation tool and used --ioengine=libaio --direct=1, bypassing both the Linux page cache and the Storage Scale pagepool. The same four workload classes were evaluated: 1 MiB sequential read and write and 4 KiB random read and write. The 1 MiB request size matched the Storage Scale file system block size; the 384 KiB request size was used only in the single-array stage, where it represented the underlying RAID full stripe.

Process counts between 8 and 96 jobs per node were tested, using workload-specific values of 8, 16, 24, 32, 48, 64, and 96. I/O depths of 1, 2, 8, 16, 32, and 64 were used as appropriate for each workload. Concurrency was increased until throughput reached a plateau or began to decline.

Each fio run used a 64 GiB working set per node and the same 200-second runtime and four-second ramp period as the raw SAN tests. Per-node directories were used in the form --directory=/gpfs/xnrfs/<test>-<node>, with --filename_format='f.$jobnum', preventing different nodes or jobs from contending for the same files. Before read or random workloads were started, all files were fully allocated and populated using a sequential 1 MiB write pass. This ensured that reads accessed real stored data rather than sparse-file holes or file system-generated zeroes.

IBM’s gpfsperf utility was used as an independent cross-check of the file system results. It employed synchronous threads rather than fio’s asynchronous queues and therefore represented applications that issue I/O from a moderate number of threads and wait for individual operations to complete. The following configurations were used on every node:

• sequential write: gpfsperf create seq -n 64g -r 1m -th 16 -dio -fsync;
• sequential read: gpfsperf read seq -n 64g -r 1m -th 16 -dio;
• random 4 KiB read: gpfsperf read rand -n 16g -r 4k -th 16 -dio -millis 20000;
• random 4 KiB write: gpfsperf write rand -n 8g -r 4k -th 16 -dio -millis 20000.

Fio and gpfsperf results were reported separately because they answer different questions. Fio measured the performance ceiling at high asynchronous concurrency, while gpfsperf represented the performance visible to a moderately threaded synchronous application. Differences between the tools were therefore expected and were not treated as measurement errors.

Stage 4 — node-failure and failback validation.

The final stage measured file system availability and performance during the simulated loss of a complete storage and compute node. Failure of node02 was simulated by stopping both GPFS and xiRAID Opus on that node. This was a controlled service-stop simulation rather than a physical power-loss test.

Before the failure, node02 owned xnraid2 and contributed one RAID member to each of the other four arrays. Its loss therefore removed the owner of one array while simultaneously degrading the remaining four arrays. The orphaned xnraid2 resource was migrated and reactivated on node03. Before performance measurements began, all five volumes were verified as available: xnraid2 was online under its new owner, while the other four arrays remained online in degraded mode.

The benchmark was then executed on the four surviving nodes: node01, node03, node04, and node05. Fio used the same direct-I/O harness, with eight jobs per node, a 200-second runtime, a four-second ramp period, a 64 GiB prefilled working set per node, and separate per-node directories. Rather than repeating the full concurrency sweep, four predefined operating points were measured:

• 1 MiB sequential read at per-node queue depth 16;
• 1 MiB sequential write at per-node queue depth 16;
• 4 KiB random read at per-node queue depth 128;
• 4 KiB random write at per-node queue depth 128.

Gpfsperf was also repeated on the surviving nodes using the same four configurations and 16 synchronous direct-I/O threads. Failure-state results were compared with the healthy five-node baseline collected earlier on the same day and on the same storage topology. After testing, node02 was returned to service, failback was completed, and reconstruction of the missing RAID members was observed until the arrays returned to a healthy state.

Concurrency selection and reporting

For the cluster-wide raw SAN and Storage Scale stages, workload concurrency was increased until bandwidth or IOPS reached a plateau. Additional concurrency beyond this point was considered counterproductive when it produced no material throughput improvement but caused a significant increase in average or tail latency.

Two operating points were identified where sufficient data were available:

• Peak performance: the tested point delivering the highest aggregate throughput or IOPS.
• Recommended operating point: the lowest practical concurrency that sustained near-peak performance while providing materially better average and tail latency.

Bandwidth and IOPS were summed across all participating nodes. The single-array stage was reported as a single-volume, single-initiator result and was not combined with cluster-wide figures. Healthy raw SAN and file system results were reported as five-node aggregates, while failure-state results were reported as four-node aggregates and compared with the corresponding healthy five-node baseline.

Average latency was calculated as the arithmetic mean of the per-node average latencies. The reported p99.5 latency was the worst 99.5th-percentile latency observed on any participating node. It was therefore a conservative worst-node measurement rather than a percentile calculated from a merged cluster-wide latency distribution.

Cache and data-allocation controls

Direct I/O was used throughout the raw SAN and file system stages to exclude the Linux page cache and the Storage Scale pagepool from the measured data path. Cache exclusion was additionally validated by monitoring node memory during large writes, comparing cold and repeated reads, and confirming that device-level IOPS reported by iostat corresponded to fio results.

Filesystem test files were fully allocated and populated before read and random workloads were executed. Together, these controls ensured that the reported results reflected the performance of the NVMe media, xiRAID parity engine, NVMe-oF transport, InfiniBand fabric, and Storage Scale file system rather than cached data or sparse-file behaviour.

GPUDirect Storage validation is documented separately because it evaluates the data path between IBM Storage Scale and GPU memory rather than the progressive storage benchmark sequence described above.

5. Results

5.1. Single-Array Raw Block Baseline

fio, libaio, iodepth=64, numjobs=16.

All 5 arrays initialise in parallel in ≈2 min; full recovery under load ≈4 min. Worst case for sizing: random write during reconstruct.

5.2. Raw NVMe-oF block-device testing

Purpose. Before any file system is layered on top, the block layer itself has to be characterised on its own terms: what the five NVMe-oF namespaces actually deliver, at which concurrency, at what average latency, including tail latency, which is important for workload suitability. This establishes the hardware ceiling of the cluster, meaning the InfiniBand fabric, the NVMe-oF transport, the xiRAID Opus RAID5 engine and the Micron media working together as one path, and it is the reference against which every layer built on top is measured.

Results — 5-node aggregate, raw NVMe-oF block devices

★ = peak throughput for that workload · ◆ = recommended operating point (best latency for the throughput delivered)

Conclusions

Random read scales cleanly with offered concurrency, and latency holds. Going from 8 to 24 processes per node raises throughput 1.6× while average latency actually improves slightly (106 → 99 µs) — at the lower process count the devices are under-driven and the limit is fio's per-process submission rate, not the storage. Pushing to 64 processes per node reaches 27.3M IOPS at 104 GiB/s, and even at that point average latency is only 191 µs. The block layer shows no sign of a hard knee within the range tested.

Random write lands at 6.55M IOPS, roughly a quarter of the read figure — the expected shape for RAID5, where every small write carries parity work in the xiRAID engine rather than being a single media operation.

Sequential throughput is bound by the fabric and back end, not by concurrency. 131.2 GiB/s read and 47.6 GiB/s write are reached already at 8 processes per node; adding depth beyond that does not raise bandwidth, it only inflates latency. The read figure is consistent with the aggregate capability of the InfiniBand front-end fabric across five nodes.

Reference value. These numbers define the hardware ceiling for this cluster and are the baseline any layer built on top of these devices should be measured against.

At the raw block level, the system delivers more than 25 million 4 KiB random-read IOPS, demonstrating high aggregate performance from the disaggregated node-local NVMe architecture.

5.3. IBM Storage Scale performance testing in SAN mode

Purpose. This section covers the performance of the xnrfs file system itself, measured in SAN access mode — every one of the five nodes reaches the NSDs directly over the NVMe-oF/RDMA fabric, with no NSD server in the data path and no I/O forwarded between nodes. All figures are for the retained 3+1 / 384 KiB array topology with a 1 MiB file-system block, and all of them are the tuned results, i.e. the state reached after the full parameter study rather than the out-of-the-box defaults. Two independent benchmark tools were used, fio as the primary instrument and IBM's own gpfsperf as a cross-check, on the principle that a single tool's saturation behaviour should not be mistaken for the storage system's.

Results — 3+1 SAN, tuned, 5-node aggregate (fio)

★ = peak throughput for that workload · ◆ = recommended operating point (best latency for the throughput delivered)

Headline tuned figures, 3+1 SAN: sequential read 131.7 GiB/s, sequential write 46.9 GiB/s, 4K random read 5.83M IOPS (5.41M at 159 µs / p99.5 255 µs for latency-sensitive use), 4K random write 3.12M IOPS.

gpfsperf cross-check (64 GiB/node, DIO, 16 threads, summed over 5 nodes)

Conclusions

Sequential throughput is fabric-bound and needs almost no concurrency. Sequential read peaks at 131.7 GiB/s with just 8 jobs per node at queue depth 16, and both tools agree on that figure. Sequential write peaks at 46.9 GiB/s at queue depth 64. Neither responds to any GPFS tuning parameter, because neither is software-limited — the constraint is the InfiniBand front-end fabric and the xiRAID back end.

5.4. Node-failure test — HA validation and degraded-mode performance

Purpose. The declustered layout means every array draws its members from four different nodes, so the loss of one node touches all five arrays at once rather than just the one it owns. That makes node failure a key availability scenario for this cluster, and it raises three questions that only a live test can answer: does the file system stay available at all, how much performance survives, and can the cluster be brought back to full health without data loss. The test was run to answer all three, deliberately as a full failure of a node rather than as a path or drive failure, because that is the case where the declustered geometry is least forgiving.

Scenario. A complete failure of node02 was simulated on the 3+1 / 384 KiB topology. The node owned array xnraid2 and simultaneously contributed one member drive to each of the other four arrays, so its loss removed one array's owner and degraded all four survivors at the same time. This represents a demanding single-node failure scenario for the declustered layout.

Procedure.

1. Fail the node. GPFS and xiRAID Opus were stopped on node02, removing it from the cluster and taking its array's owner offline.
2. Migrate the orphaned resource.
3. Verify availability before measuring. All five volumes were confirmed still present and served: xnraid2 came up online on node03, and the other four arrays ran degraded-but-online, each having lost the member that lived on node02.
4. Benchmark the surviving cluster. fio and gpfsperf were run across the four surviving nodes (node01, node03, node04, node05) using the same harness and the same operating points as the healthy-cluster reference, so that the only variable is the missing node.
5. Fail back. The arrays auto-reconstructed their missing members.

Results — 4 surviving nodes vs healthy 5-node cluster (fio)

Latency in degraded mode — the result that matters most

gpfsperf cross-check (4 nodes, 64 GiB/node, DIO, 16 threads)

Reading the results

Random I/O degrades exactly in proportion to the surviving node count, and nothing more. Four nodes out of five is 80% of the cluster, and random read retained 79.3% while random write retained 79.5%. The match is close enough that the degraded arrays are contributing essentially no additional penalty: for small random operations, the cost is simply that one fifth of the client and fabric capacity is gone. Parity reconstruction, which every read from a degraded array must perform, turns out not to bite at 4 KiB.

Sequential I/O loses an additional 10 percentage points, and that is where reconstruction shows up. Both sequential read and sequential write retained about 70% rather than the 80% that node count alone would predict. The extra loss is the bandwidth cost of reading from four degraded RAID5 arrays, where the missing member must be reconstructed from parity on every full-stripe access. This is the expected asymmetry: reconstruction consumes bandwidth, not operations, so it is invisible in the IOPS columns and clearly visible in the GiB/s ones.

Latency remained largely unchanged in degraded mode. Random read and random write average latency moved by 0.6%, which is inside run-to-run noise, and their tails moved by less than 2% (987 → 995 µs and 954 → 971 µs). Sequential average latency rose 11–14%, and the sequential read tail actually improved, from 4.05 ms to 1.42 ms, because with one node gone there is less total I/O in flight competing for the same fabric. In practical terms: an application running through a node failure sees less throughput available, but the requests it does issue come back just as fast. Degraded mode is a capacity event, not a responsiveness event.

Conclusions

The file system stayed available throughout, with zero data loss and no outage. All five volumes remained served across the failure, the benchmark, and the failback. At no point did the file system unmount, block, or lose a disk.

Roughly 70–80% of performance survives a node loss, split predictably: 79–80% for small random work, tracking node count directly, and 70% for streaming work, where parity reconstruction takes the extra cut. Capacity planning should size to the degraded figure if the workload must meet its SLA during a failure.

5.5. NVIDIA GPUDirect Storage (GDS) — enablement and testing

Purpose. GDS lets a GPU take data directly from storage into its own memory by DMA, without staging it through a CPU bounce buffer in host RAM. For a GPU-centric cluster with eight GPUsper node, that path is worth validating properly rather than assuming: the question is not only whether GDS can be made to work on Storage Scale over NVMe-oF, but whether it is actually doing direct DMA rather than silently falling back to the compatibility path that copies through the CPU and reports success anyway.

Software stack

The GDS package was deliberately matched to the driver's CUDA version (13.2) rather than taken from the newest available repository, to avoid version skew between libcufile and nvidia-fs.

Validation. gdscheck -p was run on all five nodes and returned the same result everywhere: GDS 1.17.1.22 with nvidia_fs 2.29, NVMe-oF : nvfs, compat — meaning the transport supports true GPU-direct DMA and not only the compatibility path — IBM Storage Scale : nvidia_peermem, compat, userspace RDMA supported, libcufile_rdma.so loaded, and RDMA devices configured.

Methodology

A GDS call that falls back to the compatibility path still succeeds and still reports throughput, so throughput alone is insufficient to confirm direct DMA. Two independent checks were used instead. First, gds_stats counters, which separate the two paths explicitly: on node01 in dedicated-client role, a single GPU with 8 threads at 1 MiB recorded bw=22.10 GiB/s n=124012 nvfs=124012 posix=0 — every one of the 124,012 operations took the nvfs DMA path and none fell back to POSIX. Second, an A/B against the CPU path: gdsio -x 0 (GPU_DIRECT) reached 22.1 GiB/s while gdsio -x 1 (CPU_ONLY) reached 18.3 GiB/s. GPU_DIRECT being 21% faster is the decisive evidence, because a silent compat fallback would be doing the CPU copy plus extra bookkeeping and would necessarily be slower. Together the two checks establish that the NIC is writing straight into GPU memory.

Aggregate measurement. gdsio was run in parallel on all five nodes with -x 0 (GPU_DIRECT read), -w 16 threads, 1 MiB I/O size, using one GPU per node — GPU 0 of the eight available. Files were prefilled by a CPU-side write pass first, since GPU-direct writes fail on the local path (see below). Per-node throughput was summed. Two runs were made, 15 s and 20 s, to check reproducibility.

Results

Aggregate GPFS→GPU read, 5 nodes × 1 GPU, GPU_DIRECT, 1 MiB, 16 threads:

Single-GPU true-DMA path (node01 as dedicated client):

Conclusions

GDS works end to end on this stack — IBM Storage Scale over xiRAID NVMe-oF with InfiniBand RDMA — and true GPU-direct DMA was verified by counters rather than inferred from throughput, with nvfs=124012 / posix=0 and a GPU_DIRECT path measurably faster than CPU_ONLY.

The cluster can deliver 132 GiB/s from the file system into GPU memory, using one GPU per node.

Node role must be selected according to workload requirements, as the two modes involve a performance trade-off.. A node cannot have both maximum local storage throughput and true GDS DMA. The recommendation that follows is to split roles by function: converged for pure storage and CPU nodes, where local access gives the best IOPS and GDS is not needed, and dedicated-client for GPU nodes that require true GDS.

6. Conclusions and limitations

The PoC demonstrated that NVMe drives distributed across five GPU/HPC nodes can be combined into a shared Tier-0 storage platform using xiRAID Opus and IBM Storage Scale, without a dedicated external storage array. In the tuned configuration, IBM Storage Scale delivered up to 131.7 GiB/s sequential read, 46.9 GiB/s sequential write, 5.83 million 4 KiB random-read IOPS, and 3.12 million random-write IOPS.

During the simulated loss of one node, the file system remained available and retained approximately 70% of sequential performance and 79–80% of random I/O performance. No data loss was observed during the test and subsequent failback. NVIDIA GPUDirect Storage was also validated, reaching up to 132.1 GiB/s aggregate read throughput from IBM Storage Scale to GPU memory using one GPU per node.

The current validation has several important boundaries. Node-failure testing used a controlled service-stop scenario rather than a physical power-loss event. The RAID 5 configuration is designed to tolerate a single node failure and has no margin for a second node failure until recovery is completed. In addition, the GDS results indicate a trade-off between converged storage/compute operation and a dedicated-client configuration for true GPU-direct DMA.