xiRAID Opus for Agentic and Long-Context AI Inference

Purpose and Scope

Modern AI applications often send long prompts to a language model. In agentic systems, each new request may repeat instructions, conversation history, source code, tool output, and test results from earlier steps. Before the model can start its next answer, it may need to process much of this information again. This repeated work uses GPU time, increases memory pressure, and can delay the first output token.

This research examines whether some of that repeated work can be saved outside the GPU and loaded again when it is needed. The main goal is to understand when shared NVMe storage can improve AI inference, how remote storage compares with local storage, and how xiRAID Opus can be integrated into an existing inference platform.

The research consist of two groups of tests. The first is a controlled evaluation built with Mooncake and vLLM. It compares several memory and storage configurations under short and long intervals between repeated requests. The second uses an autonomous coding-agent fleet with real tool calls and test runs. It shows how the same approach behaves when many independent agent sessions run at the same time.

Mooncake is the reference implementation used in the controlled tests. Another AI inference platform can keep its own request routing, cache management, and model-serving logic while using xiRAID Opus as a shared NVMe storage layer.

The report focuses on infrastructure behavior: request throughput, time to first token, GPU and RAM pressure, storage traffic, network use, and possible integration paths. It does not evaluate answer quality, task success, or a complete cost model.

KV Cache: Reusable Compute in Inference

Large Language Models are usually built on the Transformer architecture. A Transformer handles a request in two main stages: Prefill and Decode. During Prefill, the model reads the prompt and builds the internal state needed for attention. During Decode, it generates output tokens one at a time.

For the current token, attention creates a Query and compares it with the Keys of earlier tokens. The scores decide which Values to read. The Query belongs to the current calculation, but the Keys and Values of earlier tokens can be reused. The stored Key and Value tensors form the KV cache.

When a later request starts with exactly the same prefix, the service can load the matching KV blocks and compute only the new suffix. This does not reuse the old response. The model still produces a new response; it simply avoids repeating the preparation of an unchanged prefix (see Figure 1).

Figure 1. KV usage cases.

Without KV reuse, Prefill must process the entire prompt. If the beginning of a new request matches a prefix processed earlier, the service can load the corresponding KV data and compute only the new suffix. To support this reuse, KV data is stored in fixed-size token blocks.

Each block is identified not only by its own content, but also by the full chain of blocks that comes before it. This means that reuse is possible only for an exact matching prefix. An identical paragraph appearing in the middle of two different conversations does not produce the same reusable block. The model version, tokenizer, KV format, and parallel layout must also match; otherwise, the stored KV data cannot be reused safely.

The user-visible effect of this reuse is measured with User Time to First Token (TTFT). It starts when the request is dispatched and ends when the first output token is returned. In the controlled split architecture, TTFT includes the shared-cache lookup and load (or recomputation on a miss) followed by Prefill, Prefill-to-Decode transfer, and Decode startup. It therefore shows whether KV reuse improves the complete request path, not only the performance of the storage subsystem.

Why Agentic and Long-Context Workloads Stress Local KV Cache

A coding or research agent normally sends its conversation history on every turn: system instructions, repository or document context, earlier model answers, tool calls, test output, and the newest observation. The new part is often small compared with the repeated history.

GPU memory also has other jobs. It must hold model weights, workspaces, active requests, and active KV for current generation. Only the remaining capacity can retain older prefixes. As new traffic arrives, old blocks are evicted even if a session will need them later.

A local prefix cache is also tied to one serving replica. Sticky routing can send a session back to the same Prefill replica, but load balancing may move it to another one. A block stored only on Prefill0 is not automatically visible to Prefill1.

The important variable is reuse distance: how much other work occurs before a session returns to the same prefix. Long pauses, many active sessions, or routing changes can all increase reuse distance. When the hot tiers cannot keep the prefix for that long, an external tier can return useful state that would otherwise be recomputed.

Separating KV-Cache Intelligence from Storage Services

Efficient KV reuse needs two cooperating layers. The KV-cache manager understands the model and the serving workload. The storage layer provides shared capacity and data movement. Keeping these roles separate makes the design easier to integrate with an existing inference platform.

The KV-cache manager identifies repeated prefixes, defines stable block keys, tracks locations and validity, and keeps separate namespaces for different models, versions, tokenizers, and parallel layouts. It decides whether to load or prefetch a block, keep it in a hot tier, evict it, or repeat Prefill. It also owns routing policy and recomputation fallback.

The storage layer stores and transfers large block-oriented objects. It provides capacity that can be shared across compute nodes and can retain reusable KV after GPU memory and RAM have evicted it. It does not need to understand prompts, attention, or request routing.

The shared NVMe tier is still a cache. It is not the source of model correctness. If a block is missing, stale, incompatible, or temporarily unavailable, the serving platform treats the event as a cache miss and safely recomputes the prefix.

In our tests, Mooncake provides the KV-cache management layer and xiRAID Opus provides the remote shared NVMe tier. The same split can be used with a proprietary platform: its KV layer keeps keys, metadata, policy, and lifecycle, while Opus provides capacity, transport, RAID options, QoS, reconnect, and operational management.

Mooncake and vLLM

Mooncake is an open-source system for distributed KV-cache storage and transfer in large language model serving. In this study it provides a cluster-wide metadata service, a shared Store, clients integrated with vLLM 0.24, and an RDMA-capable transfer layer (see Table 1). vLLM runs the model replicas and exposes the serving endpoints.

Table 1. Mooncake and vLLM components used in tests.

Request flow

1. The client sends request to the gateway.
2. The gateway selects a Prefill replica.
3. Prefill checks the shared Store for reusable prefix blocks.
4. Prefill loads the reusable prefix or recomputes missing blocks.
5. Prefill transfers the prepared KV for the current request to a selected Decode replica.
6. Decode generates output tokens.
7. Decode stores newly created KV blocks for later turns.

The Store path preserves reusable context between user turns and across replicas. The Prefill-to-Decode path moves the KV needed for the current request after Prefill has loaded or computed it. In the intended flow, Prefill performs the Store GET, Decode receives the current KV through the PD connector, and Decode performs the Store PUT for new blocks.

xiRAID Opus as Shared NVMe Infrastructure for AI Inference

xiRAID Opus is a Linux user-space NVMe storage engine. It builds RAID volumes from local or network-attached NVMe namespaces and exposes them to clients over NVMe-oF RDMA/TCP or vhost-user-blk (see Figure 2).

Figure 2. xiRAID Opus structure.

Opus uses SPDK and polling in user space instead of the normal kernel block path. This matters when large KV objects must move with low latency.

Opus provides RAID, NVMe-oF initiator and target modes, partitions, QoS limits, device monitoring and reconnect, plus gRPC/CLI management. Therefore, we can create a shared NVMe tier that can scale independently from GPU servers. It exposes block storage over NVMe-oF RDMA/TCP and uses an SPDK-based user-space datapath.

Evaluation Questions and Experimental Controls

The controlled study compares six complete service configurations. It is designed to show how the preferred KV tier changes when the same agent sessions return quickly or only after a large amount of other traffic.

Questions

1. How much does local vLLM prefix caching help when a session returns to the same Prefill replica?
2. What is gained by sharing KV across Prefill replicas through a RAM Store?
3. Does an NVMe tier help after useful blocks have been evicted from RAM?
4. How does remote xiRAID Opus through CPU staging compare with local RAID0 end to end?
5. Can a direct GPU Direct Storage (GDS) path provide useful reuse without a shared RAM Store?
6. How do routing, Store loads, physical I/O, RAM eviction, and network use explain the performance results?

Test Platform, Topology, Network, and Storage

The tests run on one server with eight NVIDIA H100 80 GB GPUs and two NUMA nodes. Four GPUs are reserved for the measured service: two are independent Prefill replicas and two are Decode replicas (see Figure 3). One complete model copy runs on one GPU. Additional details are provided in Table 2.

The load generator, gateway, Mooncake services, Prefill replicas, and Decode replicas use separate CPU sets. Model processes and the shared Store are bound to the selected NUMA node. The same placement is kept for every case so that the cache or storage mode is the main controlled variable.

Figure 3. 2-Prefill × 2-Decode service and shared KV Store.

Table 2. Platform for tests.

The load generator, gateway, Mooncake services, Prefill replicas, and Decode replicas use separate CPU sets. Model processes and the shared Store are bound to the selected NUMA node. The same placement is kept for every case so that the cache or storage mode is the main controlled variable.

Model, Trace, and Workload Preparation

The workload uses Qwen3-Coder-30B-A3B-Instruct-FP8. It is a Mixture-of-Experts coding model with about 30.5 billion total parameters and about 3.3 billion active parameters per token. The experiment uses FP8 weights and BF16 KV.

The model parameters are given in Table 3.

Table 3. Model and serving parameters.

The prepared workload comes from the public codex_swebenchpro_traces corpus. It contains multi-turn coding-agent trajectories that attempt repository-level SWE-bench Pro tasks. The selected set includes 256 complete sessions and 7,903 LLM calls (see Table 4). Turn order is preserved.

Table 4. Prepared workload scale.

The next turn uses the recorded assistant response from the trace, not the live answer generated in the current run. Live output is still measured and saved, but it does not change later prompts. Without this rule, small answer differences would grow over 30 or more turns and make cache configurations hard to compare.

Compared Cache and Storage Configurations

All cases described in Table 5 retain the normal vLLM KV pool in GPU memory. The names below describe the additional reuse or storage tier.

Table 5. Compared service configurations.

Request Scheduling, Routing and Measurement Methodology

The first turn of a session goes to the least-loaded Prefill replica. The session then remembers that replica. Later turns stay on the preferred Prefill while its queued-token score is close to the best score. If it becomes much more loaded, the request spills to the other Prefill. Decode is selected independently for every request based on queued output-token budget.

This soft-sticky policy gives local prefix cache a realistic chance to help without allowing one busy Prefill to block the service. All four Prefill-to-Decode combinations carried about one quarter of the traffic. Test showed that between about 82% and 89% of non-initial turns stayed on the preferred Prefill, depending on the case.

In Experiment 1, only 16 complete sessions are active. The first group is staggered over 30 seconds. A later turn starts after the previous turn finishes and its planned start-to-start interval is reached. When one full session ends, one waiting session replaces it.

The source trace has no per-call timestamps. Experiment 1 therefore uses a fixed-seed clipped log-normal timing plan calibrated to the published inter-call distribution: mean 10.36 seconds, p50 5.16 seconds, p90 23.44 seconds, p99 80.56 seconds, capped at 300 seconds. The interval is start-to-start, so inference time is not counted twice.

In Experiment 2, all sessions belong to the scheduling population. After a turn, a session returns to the queue and normally cannot start its next turn until at least 128 other requests complete. Other eligible sessions continue to run. There is no global round barrier and no artificial think time. Near the end, the target is relaxed only enough to avoid artificial idleness.

Experiment 1: Short Reuse Interval

Experiment 1 represents a service with a limited population of active agent sessions and realistic pauses. Because sessions return relatively soon, much of the useful working set remains in shared RAM. The experiment asks whether local cache, shared RAM, local NVMe, remote Opus, or GDS best serves this hot reuse pattern. The complete results are given in Table 6 and visualized in Figures 4 and 5.

Table 6. Experiment 1 complete results.

Figure 4. Experiment 1 throughput relative to full recomputation.

Figure 5. Experiment 1 user TTFT distribution. Markers show p50, p90, and p99.

Interpretation

Local prefix cache helps the median request but not the slow tail. Throughput increases by 36% and median TTFT falls from 27.87 to 12.24 seconds, while p90 and p99 remain close to recomputation. Long or displaced prefixes are often unavailable locally.

Shared RAM is the strongest measured tier for fast reuse. The RAM-only, local RAID0, and remote Opus CPU-staging cases form a practical tie at about 1.31 requests/s and sub-second median TTFT. The local and remote storage replicas are written, but almost no physical reads occur because useful data is still in RAM.

The GDS case removes the shared 256 GiB RAM Store. It stays within 4% of the remote CPU-staging throughput and keeps a sub-second median, but p90 and p99 are higher because storage reads are directly visible in request latency.

Experiment 2: Long Reuse Interval

Experiment 2 keeps the same model, prompts, routing, and maximum 16 requests in flight, but forces each session to wait behind a large amount of other traffic. The target is at least 128 other completed requests before the next turn. This widens the working set and makes RAM eviction part of normal operation. The complete results are given in Table 7 and visualized in Figures 6 and 7.

Table 7. Experiment 2 complete results.

Figure 6. Experiment 2 throughput relative to full recomputation.

Figure 7. Experiment 2 user TTFT distribution. Markers show p50, p90, and p99.

Interpretation

Local prefix cache adds only 3.6% throughput and leaves the TTFT distribution almost unchanged. The completed Case B result confirms that sticky routing is not enough when the reuse gap is long.

RAM-only shared reuse also remains close to recomputation. It adds 9.8% throughput, while p99 stays near 65 seconds. The wider working set evicts most useful prefixes before the session returns.

Local RAID0 and remote Opus through CPU staging both reach about 0.76 requests/s and reduce p99 to about 34 seconds. Their end-to-end results are effectively identical even though the remote path includes NVMe-oF, the network, and the target-side storage stack.

The GDS path is the fastest long-reuse case. It reaches 0.931 requests/s, 14.99 seconds median TTFT, and 29.01 seconds p99. Throughput is 1.86 times the recompute baseline and 23% higher than the remote CPU-staging path, while the shared 256 GiB RAM Store is removed.

Behind the Performance Results

The headline numbers are supported by routing, Store, RAM, storage, network, and GDS records from the same runs. Figure 8 and Table 8 show that the ranking change is caused by real KV reloads rather than by shorter output, route imbalance, or a synthetic client queue.

The long-reuse RAM-only case completes 1,677 eviction cycles and reports 40.97 TB of cumulative eviction volume. Local RAID0 and remote Opus need only about 75 cycles and about 1.7 TB of eviction because the NVMe copy remains available after the RAM copy is removed. Cumulative eviction is system churn and can exceed the physical Store size many times.

In Experiment 1, local and remote CPU-backed cases write about 1.897 TiB but read almost nothing from physical storage. In Experiment 2 they each read about 34.6 TiB at an average 3.40 GiB/s and remain nearly identical end to end. The GDS case reads 41.53 TiB at 5.01 GiB/s.

The highest run-average HCA RX rate is 104.0 Gb/s, or 26% of the one 400 Gb/s receive direction, in the remote CPU-staging case during Experiment 2. GDS averages 89.2 Gb/s. HCA RX includes both Store traffic and Prefill-to-Decode transfer, so it is an upper bound on storage-only traffic.

Figure 8. Experiment 2: Store-load events, physical reads, RAM churn, and network use.

Table 8. Shared Store reuse and observed physical I/O.

Context-Length Sensitivity

The same prompt-length groups occur in every case: 1,116 requests at 32K tokens or less, 3,061 at 32-64K, 2,779 at 64-96K, and 947 above 96K. The Table 9 below show median user TTFT in Experiment 2.

Table 9. Experiment 2 median user TTFT in seconds by prompt length.

Interpretation

For the shortest group, all modes are close and GDS is slower because its fixed storage overhead is larger than the Prefill work it saves. From 32K upward, the storage-backed cases begin to separate from recomputation and RAM-only reuse.

For prompts above 96K, GDS reduces median TTFT from 52.62 seconds with recomputation to 22.10 seconds. This is 58% lower than recomputation and about 19% lower than remote CPU staging. Remote Opus remains close to local RAID0 in every longer-context group.

The observed crossover near 32K is specific to this model, KV format, hardware, and workload. It should not be treated as a universal threshold for other inference systems.

Fleet Testbed, Workload, and Measurement

The adev framework runs a fixed autonomous development loop. An architect splits the task into modules, a contract step fixes interfaces, a test writer creates checks, implementers work in parallel, an integrator assigns difficult fixes, and a gate runs pytest inside a libvirt virtual machine. A session can use up to three rounds. Each later round resends all earlier context plus failing-test text.

Tests run in one shared VM, with sessions separated by working directory. NVMe devices are created inside the guest with nvmet-loop. Only the modeled gate duration is substituted in delayed runs.

Serving uses two H100 80 GB GPUs with vLLM 0.24.0. The adev-coder and adev-reasoner roles each run Qwen3-Coder-30B-A3B-Instruct on one GPU, as its own vLLM server. Maximum context is 65,536 tokens, KV is BF16, block size is 16 tokens, and the HBM prefix pool holds 166,512 tokens per endpoint. The external Mooncake Store has a fixed 32 GB RAM tier and, depending on the mode, a local RAID0 or remote NVMe-oF disk tier below it. The remote target is the same xiRAID Opus device used in Part I.

The two studies differ in workload, GPU layout, model variant, context length, block size, RAM tier and reported metric.

Table 10. The two testbeds.

Table 11. Main axes of the agent-fleet study.

Two protocols were used. Under batch, all sessions start together and the measurement spans the whole run, so the measurements counted in the tables are independent runs. Under steady state, starts are staggered, each session relaunches on completion, and the measurements are consecutive windows cut from one run once the fleet reaches regime.

Round N begins with the full text of round N-1, so same-session prefix reuse is guaranteed by construction. Sessions share the leading task text but diverge after they generate their own material. The pause and the number of concurrent sessions work together: the pause sets how long a block must survive, while fleet width sets how quickly other traffic turns over the HBM pool.

The Store sits below the vLLM prefix cache in HBM, which Part I measures as local reuse. Writes enter the RAM tier first. Eviction can move expired-lease objects from RAM to disk. A full RAM tier that contains only live leases refuses new writes. Disk serves blocks that have already left RAM.

The reported quantity is gross endpoint busy-time saved. The metric converts KV-served tokens into avoided Prefill time using the measured Prefill rate of 14,850 tokens/s, then divides by endpoint busy time, the union of intervals in which one vLLM server has at least one request outstanding. It is not the same as end-to-end throughput improvement.

Most values are gross because Store lookup and load overhead is not subtracted. Net savings were calculated at two batch points. At one session, net is +0.005% with an interval from -0.089% to +0.098%. At six sessions without pause, gross 16.12% becomes net 13.98%, interval 8.87 to 19.09, when the read path is charged divided by the requests in flight. Charging every microsecond of it instead gives 2.98% with an interval that spans zero, but one lookup costs 4.85 ms at one session and 5.52 ms at six, so lookups do not serialize the endpoint and the divided charge is the one the measurements support. The defensible net claim at this width is 9 to 19 percent of endpoint time. Net was not recomputed for the steady-state 12- and 18-session points.

Fleet-Width Results

Figure 9. Gross endpoint busy-time saved as fleet width increases. Points use different protocols and pause settings and should be compared with that limitation in mind.

Table 12. Fleet-width results.

At one session the gross saving is 0.28% and the net figure spans zero. At two sessions the gross interval spans zero as well. The measured saving rises at wider fleet sizes and reaches 32.30% of endpoint busy time at 18 sessions, with no observed plateau. The 1-, 2-, and 6-session batch points ran without a pause; the 6-, 12-, and 18-session steady-state points ran with the 300-second pause. The two 6-session rows therefore differ in protocol and in pause and are not a controlled pair, and no single controlled curve runs across every point. The spreads of the steady-state rows bound within-run variability, while the batch intervals are taken across independent runs. The 6-session steady-state point rests on one measurement: the width sweep took a single window at each width, and only the 12- and 18-session points were re-measured over five. The sweep's own single windows at those widths, 14.65% and 30.68%, fall inside the ranges of the re-measured rows.

Why the saving grows

The external Store does not create new reuse. It serves work that HBM stops retaining. HBM hit share falls from 45.0% to 6.7%, while Store hit share rises from 38.0% to 60.7%. The endpoint capacity value of 2.54 maximum-length contexts is an indicator, not a hard threshold: at six sessions only 2.2 requests are running on average, yet saving is already 9.10% in the steady-state point.

The disk tier accounts for 9.76 percentage points of the 16.01% local result, or about 61% of the total gross saving. Without the disk tier, RAM evictions disappear instead of moving down a level. The Store serves 47-95 GiB per window rather than 150-209 GiB, and the coefficient of variation rises from 13% to 35%.

Local RAID0 and remote NVMe-oF over xiRAID Opus are close within the measured spread. Effective rates are 5.45 and 5.78 GB/s. The tier can therefore sit off-host. Both rates stand above 1.43 GB/s, the rate at which the prefix could be recomputed rather than loaded, and no medium slower than that was measured.

One remote window returned 44.02% while its hit rate fell to 35.6% against 40-50% in the others: it ran 66 preemptions against 7-17 elsewhere and issued 4,157 store keys per model call against about 1,200, so it served more by succeeding less. The four windows without that burst are the quiet windows of Table 14; they average 17.53% at a CV of 8%, against 22.83% at 52% with the burst window included.

Lease policy is critical. At a 300-second lease, the Store refuses 424.5 GiB, or 87.2%, of offered writes because live objects are not eligible for eviction. Store hit rate falls from 45.1% to 19.8%, and gross saving falls roughly by half. A short lease is therefore required for this fleet width and RAM size.

Table 13 is the width sweep, one steady window per width; its 12-session row is that single window, the one that gave 14.65%, while the per-window volumes elsewhere in this section come from the five-window disk-tier runs. Window length is recorded at two points only, 1,500 seconds in the sweep at 18 sessions and 900 seconds in the re-measured runs, so volumes per window are not comparable between the two.

Table 13. Mechanism behind the fleet-width result.

Table 14. Twelve-session operating points.

Table 15. Same-session and cross-session reuse.

Figure 10. As fleet width grows, HBM hits fall and external Store hits rise.

Figure 11. At 12 sessions, the disk tier, remote placement, and lease policy have different effects.

Cross-session reuse is only 4-5% and does not grow from 12 to 18 sessions. About 60% of touches are reuse of each session's own earlier context. The Store therefore holds the displaced tail of each session's HBM residency rather than a large shared body of content. The fleet testbed runs one endpoint per role and never moves a session between interchangeable replicas, so this measures how much content sessions share with each other. It does not measure the cross-replica case of section 3, where a session's own prefix has to be found on a replica that did not compute it.

Aged reuse means a prefix returns after the lease has expired, and it is the only class that a disk tier can serve after RAM eviction. Both captures of it are at six sessions. Without the 300-second pause it is 7.5% of block touches in one capture, and nothing recurs beyond 121.5 seconds. With the pause it is 19.5% across five runs, p99 distance is 328-443 seconds and the maximum is 330-694 seconds. The profile was not captured at 12 or 18 sessions, where the disk-tier and lease comparisons were run, and the two widths that were captured, 14.8% at two sessions and 7.5% at six, do not establish how the share moves with width. This axis is therefore less strongly measured than the fleet-width and 12-session comparisons.

The agent-fleet study shows that external-store value grows with fleet width, depends strongly on lease policy, and mainly preserves each session's own evicted context.

Study boundaries