Databases

With xiRAID, Kx improves write performance by 88% and sequential read performance by 39% compared to alternatives.

Kx Streaming Analytics

Kx provides a streaming analytics platform powered by kdb+, the world’s fastest time-series database. The platform is optimized for ingesting, analyzing, and storing enormous volumes of data including both historic and streaming data.

Kx provides a platform and a framework for designing, building and deploying data capture, processing, enrichment and analytics, as well as interactive visualizations. Solutions by Kx are used for stock market analysis, algorithmic trading, fault detection, predictive analytics, and scientific analysis, as well as for machine learning, capture of machine generated data and analytics for IoT use cases.

Solutions created on the Kx platform have extensive redundancy and fault tolerance, as well as query, filtering, alerting, reporting and visualization features. Kx has been proven to solve complex problems faster than any of its competitors.

Challenge

Kx’s columnar design and in-memory capabilities offer great speed and efficiency compared to typical relational databases. Native support for time-series operations vastly improves speed and performance of ingestion, query, aggregation and analysis of streaming and historical data.

High performance storage subsystem is a must to support very large data volumes and ensure low latency query workloads.

The solution

Kx customers use commodity server infrastructure paired with NVMe and SSD storage devices to create effective platforms that ensure high data velocity and support rich opportunities for analytics.

xiRAID is a software RAID that effectively works with SSD and NVMe drives. It is purpose-built for new types of flash drives, with a strong attention to their architecture and datapath features. That’s why NVMe based storage powered by xiRAID delivers almost full declared performance of the drives.

Kx employs xiRAID in its software R&D labs for the tests under the most demanding industrial IoT workloads and large data volumes.

The storage platform was initially tested with the following settings:

  • 124 NVMe drives 2 TB each, 48 TB RAW24 NVMe drives 2 TB each, 48 TB RAW
  • 2RAID 50 configuration with 6 RAID 5 groups
  • 3Nano benchmark running 20 threads for read and write operations

Nano benchmark calculates basic raw I/O capability of non-volatile storage, as measured from kdb+ perspective. Note, that cache is cleared after each test, unless otherwise specified.

During the test the following results were obtained:

  Before (VROC) After
Total Write Rate (async) MiB/s 3,420 6,440
Streaming Read (mapped) Rate MiB/s 14,100 19,565
Streaming ReRead (mapped) Rate MiB/s 47,905 72,461
Random Read 1 MB (mapped) Rate MiB/s 10,892 7,386
Random Read 64K (mapped) Rate MiB/s 6,992 3,542
Random Read 1 MB (unmapped) Rate MiB/s 8,914 6,485
Random Read 64K (unmapped) Rate MiB/s 5,735 3,050

Kx stores data in a columnar database, with most customer workloads involving sequential reading and writing of data. This enables Kx users to benefit from significantly higher performance in reading and writing data than available with alternative storage platforms.

Business Impact

With xiRAID, Kx has improved write performance by 88%, sequential read performance by 39%, and has lowered overall latency compared to the alternatives, enabling customers to process more data volumes with existing infrastructure.

xiRAID enables customers working with Kx technology to realize greater performance from their infrastructure investments, including:

  • collecting and processing more data with higher velocity from connected devices, sensors, and tags in the manufacturing processes and monitoring of assets;
  • capturing, storing and processing of millions of events and measurements per second to solve the most comprehensive R&D tasks;
  • delivering faster responses to the analytics and query requests for insights and machine learning applications.

About Kx

Kx is a division of First Derivatives (FDP.L), a global technology provider with 20 years of experience working with some of the world’s largest finance, technology, retail, pharma, manufacturing and energy institutions. Kx technology, incorporating the kdb+ time-series database, is a leader in high-performance, in-memory computing, streaming analytics and operational intelligence. Kx delivers the best possible performance and flexibility for high-volume, data-intensive analytics and applications across multiple industries. The Group operates from 14 offices across Europe, North America and Asia Pacific, including its headquarters in Newry, and employs more than 2,400 people worldwide.