Advanced Virtualization Performance with xiRAID Opus in Proxmox VE

Boosting virtual machine performance in Proxmox environments has always been a challenge due to inherent limitations in the existing I/O architecture. Xinnor's latest solution xiRAID Opus, a software RAID engine operating in user space, addresses this issue head-on.

Extensive experiments, conducted by Xinnor and Kioxia, have demonstrated significant performance limitations in Linux kernel-based solutions. Virtual machines are typically limited to a maximum of 308K IOPS for read operations, which is insufficient for many modern applications that require substantially higher performance. The disparity in random write performance between kernel-based solutions and bare-metal solutions is even more pronounced.

Performance further deteriorates in degraded mode, consuming more resources and severely impacting application performance. Our blog post on PostgreSQL database performance provides a detailed comparison, showing that for simple update operations, kernel-based mdraid performs 5 times slower than xiRAID Opus. In degraded mode, mdraid performance drops to 25 times slower than xiRAID Opus. These limitations make it inefficient to virtualize resource-intensive applications in environments like Proxmox using traditional methods. A common workaround involves passing multiple independent block devices into a virtual machine, which are then combined or used directly by the application. However, this approach increases management complexity, raises the probability of configuration errors, and cannot be considered truly efficient.

Our goal is to demonstrate that it's now possible to provide a single, high-performance volume to a virtual machine. This volume can be utilized by the most demanding applications, offering simplified management, reduced risk of configuration errors, and improved overall efficiency. This approach allows for effective virtualization of resource-intensive applications, overcoming the limitations of traditional kernel-based solutions and supporting high-performance applications within virtual environments.

Challenges in Proxmox Virtual Environment

In traditional Proxmox configurations, the virtio paravirtualization driver (virtio-blk, virtio-scsi, virtio-net) is commonly used to connect drives to virtual machines. However, a significant performance bottleneck exists due to the limitation of a single iothread handling the entire I/O stream for each attached drive (e.g. LVM thin volume). This restriction considerably impacts the performance of virtual machines, as increasing I/O throughput necessitates either the addition of more drives or the deployment of additional virtual machines — solutions that are often impractical for end-user applications.

Recognizing this challenge, Xinnor has developed a solution designed to overcome the performance limitations inherent in this kernel-space setup. Our approach leverages high-performance volumes that seamlessly integrate with Proxmox without replacing the standard storage options.

By utilizing the vhost interface, these volumes are exported to virtual machines through a host-based process capable of exposing virtualized block devices to QEMU instances or other processes. This innovation significantly enhances storage performance and reduces latency, ensuring that even the most demanding applications can operate efficiently within Proxmox environments.

vhost operation scheme

We’ve run 2 experiments to test the following:

1. Theoretical performance of 1 virtual machine with resources given to one vhost-target.
2. Performance scalability across a range of 1 to 8 virtual machines connected to vhost-targets.

These experiments aim to validate the performance gains and scalability improvements offered by our solution in Proxmox environments.

About Proxmox VE

Proxmox Virtual Environment is a complete, open-source server management platform for enterprise virtualization. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage, and networking functionality, on a single platform. With its integrated web-based user interface you can manage VMs and containers, high availability for clusters, or the integrated disaster recovery tools with ease.

About xiRAID Opus

The xiRAID Opus software implements a storage system using a user-space data path engine and management components. The user-space engine enables the system to achieve high speeds and eliminates dependencies on the operating system version, drivers, and libraries.

The xiRAID engine is a high-performance software RAID system, leveraging multiple components to optimize performance. One key component is the internal userspace high-performance hardware drivers, facilitating efficient IO operations with local or network-connected devices. By bypassing the OS drivers, xiRAID interfaces directly with the hardware storage devices. To use a device with the storage engine, it must be detached from the operating system and attached to the xiRAID engine for visibility. Once attached, the device is managed exclusively by xiRAID and is no longer accessible in the host OS. Upon restart, xiRAID automatically re-attaches previously connected storage devices.

Additional information about xiRAID Opus can be found here:
https://xinnor.io/resources/xiraid-opus/

Testing environment

• Platform: Supermicro server AS-2125HS-TNR
• CPU: 2х Genoa 9454 DP/UP 48C/96T 2.75G 256M 290W SP5
• RAM: 256GB DDR5 Memory
• Drives Subsystem: 10 x Micron_9400_MTFDKCC7T6TGH
• Proxmox version: proxmox-ve: 8.2.0 (running kernel: 6.8.8-1-pve)
• xiRAID Opus Version: 1.0.1
• RAID configuration: RAID 6 with 10 drives ss=128 bs=4096

Test stand architecture

Storage subsystem backend performance

To obtain basic values of the disc subsystem performance and to make sure that there are no bottlenecks on particular hardware, we perform backend measurements with fio-3.33.

The result of measurements is presented in the table below, the link to the configs can be found in the appendix. As you can see from the table, the drive's performance is close to the performance declared by the manufacturer.

Proxmox and xiRAID Opus configuration

For testing, we used a RAID 6 with 10 drives with a 128kb strip size created in user space. In case of testing scalability of 8 virtual machines performance, RAID was divided into 8 partitions to distribute the RAID resources among virtual machines. A vhost-target was created on each partition. Each vhost-target was allocated 2 cores in polling mode from the subset of 16 allocated xiRAID Opus cores.

In case of testing 1 virtual machine, 1 vhost-target was created for the entire RAID volume and all 16 cores were allocated to this vhost-target.

Proxmox was installed with default parameters.

The output of the RAID configuration parameters

Installation and configuration steps

Below is an example of the steps to configure xiRAID Opus and virtual machines to test the scaling of I/O operations on 8 virtual machines.

For the tests on one virtual machine, we skipped point 4 with partitioning and created 1 vhost with CPU bitmask 0xFFFFFFFF.

1. Download and install xiRAID Opus

Download xiRAID Opus and install it according to the instruction: https://xinnor.io/resources/xiraid-opus/. 0xFFFF means 16 cores allocation for xiRAID Opus and 64 GB RAM.

2. Prepare the drives

Connecting the drives to xiRAID Opus user space.

3. Create RAID

Creating RAID6 named “media6”, chunk size equals 128, block size equals 4096.

4. Creating partitions

RAID was split into 8 partitions, -s describes the partition size in MB.

5. Creating vhost device for each partition

vhost target is created for each partition (media5p0, media6p0, media6p1, media6p2, media6p3, media6p4, media6p5, media6p6, media6p7). 2 CPU cores allocated for each vhost target.

To achieve high performance, we assign a specific CPU mask to each controller. It is preferable that these CPU masks for different controllers do not overlap. This approach ensures optimal resource allocation and helps prevent performance bottlenecks.

6. Connecting partitions to VMs

At this stage, you need to edit the virtual machine configuration file. This is done using a text editor, not through the Proxmox UI, as there isn't currently an option to connect vhost-targets directly from the interface.

Edit the VM configuration file. For example, if your VM ID is 100:

Add the following arguments to the file:

Argument breakdown:

1. Memory backend specification: -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem This specifies the shared memory for the VM. QEMU must share the VM's memory with the SPDK vhost target.
2. Vhost-BLK device addition: -chardev socket,id=char1,path=/opt/xiraid/bin/xnr_conf/sock/vhost.X -device vhost-user-blk-pci,id=blk0,chardev=char1 This adds the Vhost-BLK device to the VM.

Important notes:

• The size option must match the amount of memory dedicated to the VM.
• Each VM will have its own vhost controller with a unique name.
• Replace vhost.X with the number of the vhost controller you created for this specific VM.
• Ensure you use the correct vhost controller number for each VM to avoid configuration errors.
• After configuration, vhosts will be available in VMs as block devices: /dev/vda

Testing Linux Virtual Machines

Testing full capacity of 1 VM

Performance testing was performed from virtual machines using fio-3.36.

Virtual machine configuration:

• Ubuntu 24.04 LTS (6.8.0-35-generic)
• CPU cores: 16
• RAM: 16384 MB

fio read/write config:

fio random read/write config:

Results

This document uses the following baseline performance metrics, based on a 10-drive backend configuration:

• Random read: 100% of raw backend performance (16M IOPS)
• Sequential read: 100% of raw backend performance (72.5 GB/s)
• Random write: 33% of raw backend performance (1.22M IOPS). Calculation: 3.66M IOPS x 33% = 1.22M IOPS
• Sequential write: 80% of raw backend performance (55.4 GB/s). Calculation: 69.3 GB/s x 80% = 55.4 GB/s

The reduced write performance percentages (33% for random and 80% for sequential) reflect the overhead introduced by the RAID6 configuration.

Our sequential read and write performance tests have achieved speeds of several tens of GB/s. This level of performance enables the use of virtual machines for demanding tasks that require such high throughput, including HPC and AI applications. Essentially, we can now build solutions for modern, data-intensive workloads with storage that can reach tens of gigabytes per second within a single virtual machine.

(*) In random write operations, we have exceeded the baseline performance. This is due to the user space drivers employed by Opus, which operate in polling mode. These drivers are significantly more efficient than those working in kernel space. This efficiency has allowed us to achieve RAID performance that surpasses the baseline measured in kernel space, resulting in lower latency, higher performance, improved efficiency, and reduced CPU load.

Regarding specific metrics:

1. Random Write performance exceeded our expectations.
2. For Random Read, while we didn't reach the baseline, this was due to the limitation of 16 cores in the virtual machine. Nevertheless, we achieved an overall performance of 6.2M IOPS, comparable to high-end, expensive storage solutions.
3. We attained approximately 400K IOPS per core, whereas kernel-based solutions typically achieve 300K IOPS across all 16 cores. This demonstrates a nearly 20-fold increase in efficiency: 300K IOPS for 16 cores versus 400K IOPS for a single core.
4. It's important to note that these results were achieved using a single device. To obtain similar figures using third-party solutions, even with a competing high-performance RAID solution, one would need to deploy 20 block devices within a virtual machine - a setup that would be impractical in real-world scenarios.

Testing multiple VMs

Performance testing was performed from virtual machines using fio-3.36.

Virtual machine configuration:

• Ubuntu 24.04 LTS (6.8.0-35-generic)
• CPU cores: 4
• RAM: 8192 MB

fio sequential read/write config

Results:

Our tests with multiple VMs for sequential write operations demonstrate nearly linear scalability when using small, low-footprint virtual machines. We observed that adding more VMs results in a proportional increase in performance. Under certain configurations, we can achieve backend performance levels, effectively maximizing storage utilization.

fio sequential randread config

fio sequential randwrite config

Results:

For random workloads, we observe similar scalability:

1. Random write: performance scales with the addition of small virtual machines, meeting and even exceeding the expected baseline.
2. Random read: we achieve performance levels comparable to high-end storage solutions. While we don't reach backend performance, this is primarily due to the limited number of cores in the virtual machines used for testing.

It's important to note that increasing the resources allocated to virtual machines could potentially yield performance in the range of tens of millions of IOPS. For reference, a previous blog post demonstrated achieving approximately 23M IOPS in a similar configuration.

These results indicate that our solution offers excellent scalability and performance across various workload types, with the potential for even higher performance given more powerful virtual machine configurations.

Conclusion

There are significant limitations when using traditional kernel-based solutions for storage management:

1. Performance bottleneck. There's a constraint on performance per iothread. This necessitates the use of multiple volumes for high-performance applications, which is both inconvenient and potentially non-functional in some scenarios.
2. Degraded mode performance. When using Linux RAID, performance in degraded mode severely impacts real-world application performance, as detailed in one of our previous blog posts.

We have now demonstrated that our solution xiRAID Opus not only works with virtual machines but also integrates seamlessly with Proxmox. xiRAID Opus enables a single virtual machine to achieve performance levels comparable to a large physical server connected to a high-end storage system. This level of performance is typically unattainable with competing solutions, especially within a virtualized environment.

In summary, xiRAID Opus addresses the limitations of kernel-based solutions while offering superior performance and flexibility, particularly in virtualized environments like Proxmox VE.

Appendix

Backend measurements config

Show content

VM Configuration example

Show content

Useful links

https://kvm-forum.qemu.org/2023/Multiqueue_in_the_block_layer_wLom4Bt.pdf

https://www.smartx.com/blog/2022/07/spdk-vhost-user-en/

https://spdk.io/doc/vhost.html

https://events19.lfasiallc.com/wp-content/uploads/2017/11/Storage-Performance-Tuning-for-FAST-Virtual-Machines_Fam-Zheng.pdf

https://spdk.io/doc/vhost.html#vhost_qemu_config