This paper outlines the process of configuring a high-performance Network File System (NFS) storage solution using the xiRAID RAID engine, Remote Direct Memory Access (RDMA), and the XFS file system. Modern data-intensive workloads (such as those in AI & machine learning, high-performance computing, scientific research, media & entertainment (e.g., 4K/8K video rendering, real-time asset streaming), virtualized environments requiring rapid storage access, etc.) demand storage subsystems capable of delivering extreme throughput with minimal latency.
By leveraging xiRAID’s optimized RAID engine alongside RDMA’s low-latency data transfers and XFS’s scalability, this approach achieves unprecedented sequential access performance, critical for large-scale datasets, while offering actionable insights for improving random read/write efficiency.
The document focuses on maximizing throughput and reducing latency, particularly for sequential access patterns common in scenarios like AI/ML model training, large-scale HPC simulations, real-time media rendering pipelines, virtualized infrastructure requiring consistent I/O performance, etc. Though it does not cover full production configurations (e.g., security settings), the procedures outlined enable organizations to deploy a high-performance NFS storage foundation that balances simplicity, scalability, and raw speed.
InfiniBand Network Setup
For RDMA access, both the NFS server and clients must have the NVIDIA MLNX_OFED driver support (--with-nfsrdma) installed. Download the appropriate driver for your operating system from NVIDIA’s website and install it using the following command:
Ensure that similar configurations are applied on the NFS clients. Configure the InfiniBand adapters and verify the network settings. The following utilities can be used to check the performance of the Infiniband network:
- ib_send_bw for bandwidth testing.
- ib_send_lat for latency testing.
- ib_read_bw and ib_read_lat for RDMA read bandwidth/latency.
- ib_write_bw and ib_write_lat for RDMA write bandwidth/latency.
Disk Subsystem Performance Check
Before setting up RAID, configuring a file system and NFS server, determine the desired RAID level for data and file system journals.
Test raw drive performance to ensure no bottlenecks exist at the server or disk subsystem level. Refer to Xinnor's performance guide for detailed recommendations:
https://xinnor.io/blog/performance-guide-pt-2-hardware-and-software-configuration/
After ensuring that the drives performance meets the requirements, start setting up RAID and configuring the file system.
xRAID Setup
This document uses RAID 6 with 10 drives and a strip size of 128k for data, and RAID 0 with a strip size of 16k for file system logs. (In production environment it is better to use RAID 1 or 10).
Install the latest xiRAID version using the documentation and create the RAID arrays as follows:
xicli raid create -n media6 -l 6 -d /dev/nvme16n2 /dev/nvme9n2 /dev/nvme20n2 /dev/nvme18n2 /dev/nvme8n2 /dev/nvme12n2 /dev/nvme13n2 /dev/nvme19n2 /dev/nvme23n2 /dev/nvme24n2 -ss 128 xicli raid create -n media0 -l 0 -d /dev/nvme7n1 /dev/nvme6n1
Check the RAID status after the initialization process is complete using xicli raid show.

After creating the RAID array, verify that the performance at the RAID layer meets expectations before proceeding to file system creation. For RAID performance checks and additional tuning, refer to the materials in Xinnor's performance guide.
Based on our testing experience, the sequential write speed to the RAID should be approximately 90-95% of the write speed to raw data drives.
XFS Setup and Mount
Create the XFS file system with the following command:
mkfs.xfs -d su=128k,sw=8 -l logdev=/dev/xi_media0,size=1G /dev/xi_media6 -f -ssize=4k
Depending on the geometry of your RAID, the file system creation options may vary. Pay special attention to parameters such as su=128k and sw=8, as these are important for aligning the file system geometry with the RAID configuration.
Parameter | Description |
---|---|
-d su=128k,sw=8 |
-d: Refers to the data section of the filesystem.
|
-l logdev=/dev/xi_media0,size=1G |
-l: Refers to the log section of the filesystem.
|
/dev/xi_media6: | The xiRAID block device on which the XFS filesystem is being created. |
-f | Forces the creation of the filesystem, even if the target device already has data or a filesystem. |
-s size=4k | Specifies the sector size for the filesystem. In this case, it is set to 4 KB, which matches most modern disk configurations for optimal performance. |
Mount the file system using:
mount -t xfs /dev/xi_media6 /mnt/data -o logdev=/dev/xi_media0,noatime,nodiratime,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k
Parameter | Description |
---|---|
logdev=/dev/xi_media0 | Specifies the dedicated log device for the XFS journaling system. Using a separate, fast log device reduces the latency of journal operations, which is particularly beneficial for write-heavy workloads. |
noatime | Prevents updating the access time of files when they are read, thereby reducing write operations and improving performance. |
nodiratime | Prevents updating the access time of directories, further minimizing write operations. |
logbsize=256k | Sets the log buffer size to 256 KB. Larger log buffer sizes can improve performance for workloads with high transaction rates. |
largeio | Optimizes performance for large I/O operations. This works well with the allocsize parameter to handle large files effectively. |
inode64 | Allows allocation of inodes anywhere in the filesystem, enabling support for large filesystems (greater than 1 TB). Recommended for modern systems with large storage pools. |
swalloc | Optimizes allocation for striped RAID arrays by ensuring alignment with the RAID stripe. |
allocsize=131072k | Sets the default data allocation size to 128 MB. This is beneficial for workloads with large sequential writes, such as media storage or backups. |
Similar to the RAID setup, you should also test the performance of the file system by writing several large files to a directory. The performance should be approximately 70-80% of the RAID performance for writes and 90-100% for reads.
For permanent mounting of the file system, follow the recommendations in the xiRAID documentation.
NFS Server Setup
With the disk subsystem, RAID, and file system properly configured, the next step is to install and configure the NFS server. The installation is straightforward, but several optimizations are necessary to achieve better performance.
1. Install nfs-utils packages
2. Firewall setup. Below is an example of a simple firewall configuration for a test environment:
firewall-cmd --permanent --add-service=mountd
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --reload
3. NFS share directory creation. The following commands create a directory for an NFS file share. These settings are suitable for testing purposes only, as they do not include access restrictions:
chown nfsnobody:nfsnobody /mnt/data
chmod 777 /mnt/data
4. NFS export configuration. For the /mnt/data directory, typically found in /etc/exports. If the file does not exist, create it manually and add the following line in the /etc/exports file:
Parameter | Description |
---|---|
/mnt/data | Specifies the directory to be exported (shared) via NFS |
* | Allows access to all clients, regardless of IP address. In a production environment, it’s recommended to replace * with a specific client IP, hostname, or subnet (e.g., 192.168.1.0/24). |
rw | Grants read/write access to the shared directory. Clients can read from and write to /mnt/data. |
sync | Ensures data is written to disk before confirming a write operation. |
insecure | Allows connections from ports above 1024. |
no_root_squash |
Prevents the "root squashing" mechanism, which maps the root user on the client to the nobody user on the server. This allows the root user on a client to act as root on the NFS server, which can be a significant security risk. |
no_subtree_check | Disables subtree checking, which verifies that a file accessed by the client is within the exported directory. This improves performance but reduces security slightly. |
no_wdelay | Disables write delay, ensuring that write operations are performed immediately. This can improve performance in certain scenarios. |
5. NFS Server configuration tuning. Edit the NFS server configuration file /etc/nfs.conf to adjust the number of threads for handling requests and enable RDMA connections. The configuration should resemble the following:
# debug="all|auth|call|general|parse"
# manage-gids=n
# state-directory-path=/var/lib/nfs
threads=64
[nfsd]
# debug=0
threads=64
# host=
# port=0
# grace-time=90
# lease-time=90
# udp=n
# tcp=y
vers3=y
vers4=y
vers4.0=y
vers4.1=y
vers4.2=y
rdma=y
rdma-port=20049
Parameter | Description |
---|---|
[exportd] threads=64 |
Specifies the number of threads used by the exportd daemon, which handles the export of NFS file systems. This setting determines how many threads will be used to handle client requests for mounted directories. 64 threads allow a moderate number of clients, but this can be adjusted based on the system’s performance needs. |
[nfsd] threads=64 |
Specifies the number of threads used by the nfsd daemon, which serves NFS requests. This defines the number of worker threads the NFS server will use to handle incoming NFS client requests. Like the exportd thread setting, this is crucial for performance and scalability. Adjusting this number will impact how many simultaneous client requests can be processed. |
rdma=y rdma-port=20049 |
Enables Remote Direct Memory Access (RDMA) for NFS operations uses port 20049 |
6. Enable, restart and check NFS Server
After applying all settings, enable and restart the NFS server:
systemctl restart nfs-server
Check the status of the NFS server to ensure there are no errors, particularly those related to the RDMA module:
NFS Client Setup
You can now proceed to configure the NFS client.
1. Install nfs-utils packages
2. NFS client kernel module options.
Add the following line to /etc/modprobe.d/nfsclient.conf:
Increasing the max_session_slots value allows more simultaneous in-flight requests, improving performance for workloads with many small or parallel I/O operations.
3. Reboot the system
4. Mount NFS Share
Create a directory for the NFS file share (in this example, /mnt/nfs) and mount it using the following command:
Parameter | Description |
---|---|
rdma | Enables Remote Direct Memory Access (RDMA) for NFS. |
port=20049 | Specifies the port to use for the NFS service. The port 20049 must be open on the NFS server and mapped to the NFS service |
nconnect=16 | Opens 16 connections to the NFS server, improving parallelism and potentially increasing throughput. |
vers=4.2 | Specifies the NFS protocol version to use |
After completing the setup, you can run performance tests from the NFS client. If everything is configured correctly, the performance should meet expectations. Use fio with the following configuration for testing:
rw=read
#rw=write
bs=1024K
iodepth=32
direct=1
ioengine=libaio
runtime=4000
size=32G
numjobs=4
group_reporting
exitall
[job3]
directory=/mnt/nfs
Depending on the configuration, you can expect NFS performance to be 50-70% of the XFS file system's performance.
Conclusion
By following the outlined steps, users can set up a high-performance NFS storage system leveraging xiRAID and RDMA. The configuration ensures optimal performance for sequential data access patterns and provides flexibility for tuning based on specific workload requirements. For production environments, additional configurations such as security settings should be applied as needed.