Storage has tiered for decades: hot data where access is fast, cold data demoted to cheap capacity, promoted back on demand. The same argument now applies to GPU memory — the cold data is an AI agent’s paused session.
When a session pauses, the GPU throws away the work it just did
Coding agents are the sharpest case for this. Their context only grows — every turn resends the task, the files read so far, the code written and the test output being fixed — and between turns they stop, sometimes for minutes, while tests run or an engineer reviews the output. They combine the largest per-session state with the longest idle gaps: the most to lose from HBM, and the most expensive to rebuild. Nothing here is a new technique. It is a workload shape, and it is the shape that decides whether a tier below HBM is worth having.
An agent resends its whole conversation every turn, and the model rereads all of it before writing a single token. That reread is prefill — the bulk of an agent fleet’s GPU work. It produces the KV cache, the state kept for every token read; while the matching prefix remains in HBM, the service can avoid most of the repeated prefill work.
HBM is small and every session shares it, so a paused session’s cache is evicted when others need the room. With nothing underneath, the state is discarded rather than demoted, and resuming means reading the conversation again at full GPU cost.
Give it a tier and the session can be frozen instead, restored on its next turn without the GPU recomputing anything. It is hibernating a laptop, except the memory written out is the GPU’s.
None of it is new, but it takes two pieces the serving stack does not ship with. Above, a KV store plugged into vLLM decides what to keep and when to let it fall: we ran Mooncake. Below, the capacity: we ran xiRAID Opus, serving NVMe over the fabric. The drives are not the constraint — rebuilding that state takes longer than reading it back, from RAM or from NVMe.
One difference from classic tiering is worth naming. A storage tier demotes because the data went cold. This one demotes because another session needed the room, whatever the age of what it holds.
The capacity-price gap decides what the tier is made of
Traditionally, RAM holds operational state while storage systems hold persistent data. KV cache fits neither category: it is transient by nature, but at scale it needs storage-like capacity and infrastructure. That raises an architectural question — should KV cache reside entirely in RAM, or can it reuse high-performance enterprise storage infrastructure that has already been deployed and paid for?
Scaling KV capacity with memory alone is economically unsustainable. On a capacity basis, OEM NVMe is roughly 40–50× less expensive per terabyte than OEM DDR5.
Source: Supermicro public OEM list prices (online configurator), retrieved August 2026 — enterprise NVMe ≈ USD 737–884 per TB; DDR5 ≈ USD 41,000–42,000 per TB-equivalent. We quote 40–50× rather than the literal ratio of those list prices, since volume and street pricing compress the gap.
A purchased enterprise array is the other thing teams reach for, and it fails the same test at a smaller multiple. At roughly USD 2,000–4,000 per TB it costs some 3–5× the raw OEM NVMe it is assembled from, and it adds a separately scaled box beside GPU nodes that already hold qualified NVMe.
This leads to a second question: do we need large cache capacity at all, or would a few gigabytes of very fast RAM be enough? However fast a storage layer is, matching DRAM latency remains extremely difficult, and if a small RAM cache already covers the effective working set, a much larger storage tier adds little. The question is therefore not whether storage can hold KV cache, but how large the effective working set has to be before capacity matters more than raw memory performance.
Micron’s analysis puts the two knees at different scales. For one VR200-class GPU the reference performance threshold is approximately 12 GB/s of tier bandwidth; beyond it, additional bandwidth gives diminishing returns unless the rack-level tier is large enough to retain reusable KV data. For one modelled 72-GPU rack, the capacity knee is approximately 820–850 TB.
Source: Micron, Optimizing KV Cache Offload for Scalable, Cost-Efficient AI Inference, FMS 2026, slide 3. The slide states a knee below approximately 850 TB; this brief uses approximately 820 TB as the rack design reference. Both values are model- and workload-dependent.
Micron’s 1.6-million-configuration sweep estimated five-year NVMe TCO savings of 43–96%, depending on endurance and write-amplification assumptions, with positive ROI in approximately 99.6% of dense-model configurations at production request rates.
Three properties make a shared NVMe pool the right shape for this tier.
- Sharing. RAM is tied to a specific server or enclosure and is difficult to share efficiently across nodes. Shared-memory approaches are technically possible but add significant architectural complexity, while a dynamic NVMe pool distributes across compute nodes easily and scales independently of the servers themselves.
Pooled memory over CXL is the obvious objection here, and it narrows the wrong gap. What ships today is host-attached expansion: CXL 2.0 switches are in production and the first CXL 3.0 switches only began sampling in 2026, and the fabric-attached pools built around them are sized in tens of terabytes against a rack-level working set of hundreds. CXL memory is also still DRAM — its cost per terabyte tracks DDR5, not NVMe — so pooling moves where the capacity sits without changing what it costs. That makes CXL a latency play, and latency is not what constrains this tier. It is worth revisiting when switched pools are qualified inside GPU nodes at rack scale.
- Uncertainty. Both the workload profile and the required KV cache capacity are inherently difficult to predict. They depend on model evolution, context length, concurrency, serving patterns and future application requirements, which makes a DRAM-based cache infrastructure hard to size efficiently.
- Minimum working set. A cache only works above a minimum working set. Below it the hit rate collapses and more bandwidth buys nothing, because the binding constraint has become capacity. That threshold is what the rest of this brief is sized against: Micron puts the rack-level capacity knee at roughly 820–850 TB for a modelled 72-GPU rack. At that size a DRAM tier is not a budget question but a non-starter — which is precisely what makes it an NVMe tier.
Given these factors, NVMe is the practical and cost-efficient way to build a large, scalable and dynamically shared cache tier.
When the lower tier actually gets used
First, how long the session stays cold decides which tier holds it — less the clock than how much other traffic passes through meanwhile. Short gaps, RAM serves everything; long gaps, only NVMe still holds the session.
Second, nothing moves to the lower tier until something needs the room. The GPU asks the KV store only after it has evicted the matching KV from HBM. Inside the store, KV reaches NVMe only when the RAM tier needs space and the object’s lease has expired. An agent working alone may therefore never trigger enough eviction for the lower tier to matter.
With one or two sessions on a GPU, nothing was evicted, so the tier had nothing to give back — even though those GPUs were running flat out. As the number of sessions sharing a GPU grows, eviction begins and the share of prefill work the tier takes off the GPU grows with it; at the widest fleet we ran, it was still growing. Both figures come from the store’s own counters, not from a run with the tier off. So what decides whether the tier pays is not how busy a GPU is; it is how many sessions share it.
Third, the tier fills with one stream per session. When a group of agents runs, what reaches the tier is not one shared working set: each agent re-reads far more of its own earlier context than of anything it has in common with the others. The tier fills with as many separate streams as there are sessions, and capacity is sized against every session in flight rather than against the smaller part they share.
Nor is all of it cacheable. Roughly one lookup in three was for something never read before — a new file read, a tool output just produced. No tier can serve that; it has to be computed, whatever sits below. A miss is not a failure mode: the model simply reads the conversation again, exactly as it does today without a tier. The turn is slower, and nothing breaks.
One configuration mistake can switch the tier off entirely
The cache manager above decides what actually reaches the tier. A lease (--default_kv_lease_ttl in Mooncake) pins each written object until it expires, and nothing can be demoted before then. The lease must therefore be shorter than the tier’s capacity divided by the write rate. Longer, and nothing is evictable: protected objects hold the RAM tier, new writes are refused, the work is recomputed on the GPU instead, and the disks sit idle. Ours was longer. It cost about half the saving.
What Xinnor validated, and why the tier should be shared
Xinnor’s validation ran a real agentic workload rather than a synthetic prefix-sharing benchmark: coding-agent sessions with growing context and long idle gaps, served through vLLM with Mooncake as the KV manager and xiRAID Opus underneath as the storage tier. The KV manager owns the cache logic — it identifies reusable blocks, decides when they may leave RAM, and falls back to prefill on a miss. What we validated is what happens below it: whether a shared NVMe tier reached over the fabric returns the state fast enough to beat recomputation, and whether sharing costs anything against drives sitting inside the GPU server.
The chart reads left to right as layers added under the same workload, each normalised to full recomputation.
- Full recomputation, 1.00×. The baseline: no reuse, every turn rebuilt on the GPU.
- Local vLLM prefix cache, 1.04×. By the time these sessions return, enough other traffic has passed that the matching prefix is no longer in HBM. Prefix caching inside the server does almost nothing for this workload.
- RAM store only, 1.10×. The RAM tier is too small to hold the cold set once sessions multiply, so most returning sessions still miss.
- RAM store + local RAID0, 1.52×. Adding NVMe underneath is where the gain comes from. What the tier supplies is capacity, not speed.
- RAM store + remote xiRAID Opus RAID0, 1.52×. The architectural result: remote over the fabric matches local drives exactly. Sharing the tier across nodes costs nothing measurable, so the capacity does not have to be replicated inside every GPU server.
- xiRAID Opus RAID0 + GDS, 1.86×. Without the RAM store at all. Removing CPU staging from the path is worth more than the RAM tier it replaces.
Three conclusions follow. The drives are not the constraint: rebuilding KV state on the GPU takes longer than reading it back, from RAM or from NVMe, which is what leaves room for a tier at all. What the tier is worth is decided by sessions per GPU, not by GPU utilisation. And because reuse is dominated by each session’s own earlier context, the cold working set grows with the number and length of active sessions — held inside each GPU server, that capacity has to be provisioned server by server.
xiRAID Opus composes it into a shared NVMe pool instead, one that scales separately from GPU compute and is reachable from several compute nodes — and, as the chart shows, at no throughput cost. Its SPDK-based user-space datapath exposes RAID volumes over NVMe-oF RDMA/TCP. RAID, QoS, reconnect, monitoring and central management make the lower tier shared infrastructure rather than a set of local disks attached to individual GPU servers.
- Composes a protected pool from devices already installed. Opus disaggregates OEM-qualified NVMe in GPU and other server nodes and composes it into a shared software-defined RAID pool. This reclaims dark flash, removes individual-host capacity boundaries and reduces the need for a separately purchased enterprise-storage tier.
- Preserves node resources for AI workloads. The user-space architecture is designed for a small CPU and DRAM footprint, leaving as much host capacity as possible for model serving and KV-cache indexing and search engines. Exact overhead should be validated on the target hardware and topology.
- Redirects capacity while maintaining predictable service. Local and network-attached NVMe can be composed around current demand, while end-to-end QoS isolates latency-sensitive reads from sustained mixed read/write traffic and prevents one workload from consuming the entire tier.
- Protects the shared hot set. System-prompt KV is highly reused and has the widest failure blast radius: losing one region can affect many sessions at once. Opus can place it in the fastest pool with explicit redundancy and priority, while session KV uses a larger elastic pool.
- Enables a GPU-direct storage path. The architecture is GDS-ready, allowing validated configurations to reduce CPU-mediated data movement between storage and GPUs.
What we do next
- Productise the GDS path for Mooncake. The 1.86× is a prototype result on a single configuration; the next step is measuring it across the fleet sweep.
- Re-run the sweep with a correctly sized lease, to recover the roughly half of the saving the misconfiguration cost.
- Extend the sweep past the widest fleet we ran. The tier’s share of prefill work was still rising there, so the ceiling is not yet known.
- Measure the protected system-prompt pool and QoS isolation under mixed read/write load, rather than reasoning about them.
The full study report is at https://xinnor.io/blog/xiraid-opus-for-agentic-and-long-context-ai-inference/.
