For decades, Lustre has been the storage backbone of the world's largest supercomputers — Frontier, Aurora, Leonardo all depend on it. Built for sustained sequential throughput at national-lab scale, it excels at the workloads that defined HPC: big simulations, big writes, big reads, and lots of compute.
But the defining AI workload has shifted. Inference has overtaken training as the dominant workload in AI infrastructure, and it looks nothing like training. Where training is a long, write-heavy batch job, inference is a continuous stream of latency-sensitive requests, each reading from shared state and producing real-time output for a real user — many small reads, long context, and persistent state. The storage profile is fundamentally different.
At the same time, a growing number of enterprises want to run these workloads on premise, on hardware they control, with the same multi-turn experience their users get from public chatbots. That means solving the full storage problem locally: KV cache, session state, retrieval — without offloading any of it to the cloud.
So, can Lustre solve the storage bottlenecks of multi-turn inference for enterprises running entirely on premise? This post documents what we found. We start with why storage has become a first-class concern in AI inference pipelines — the KV cache, the agent loop, and the shift to disaggregated serving architectures. We then walk through our implementation, which integrates Lustre's read-only Persistent Client Cache into the vLLM/LMCache stack on top of xiRAID storage. Finally, we present throughput and latency results from our MLPerf Storage test sweep, and an offline eviction policy study that helps explain what makes PCC effective — and when a different policy would be the right choice.
Part 1. Why storage for AI inference has become a hot topic
Here is the basic shape of an AI inference pipeline. A user sends a request. The system retrieves context — a prompt, retrieved documents, maybe chat history. Model weights and retrieved data both come from shared storage. The GPU runs the inference: attention math, token decoding, output.
The critical change is in the KV cache. KV stands for "key" and "value" — these are the intermediate tensors computed by each attention layer. The model reuses them on every generated token. In a long conversation, this cache grows large. Too large for GPU HBM alone. So, it spills: to DRAM, to local NVMe, and finally to shared network storage.
Storage is no longer just a place to load model weights from once at startup. For long-context, multi-turn, and disaggregated inference, it becomes part of the context-memory hierarchy: reused or offloaded KV state may be fetched from shared storage when it cannot remain in GPU HBM.
The agent loop makes it worse
A single-turn chatbot is already demanding. An agent is categorically harder. Agents plan, call tools, read documents, write drafts, and run in loops until a task is complete. Each cycle adds more state: tool outputs, intermediate reasoning, retrieved data, verifier feedback — all of it flows into the next prompt.
The working set grows fast. A simple chat might use 10,000 tokens of context. An agent working on a real software project can exceed 200,000 tokens. The KV cache and session state cannot live in GPU memory alone. They need a tiered storage hierarchy — HBM for the hottest data, DRAM for warm, NVMe for cooler, shared storage for everything else.
Prefill, decode, and the prefix cache
Three terms appear throughout this post, so let us define them clearly: prefill, decode, and prefix cache.
When an LLM receives a request, the prompt is first split into tokens. The model then processes those tokens through its transformer layers. For every token, at every layer, the model computes internal attention data: keys and values, usually shortened to K/V. These K/V tensors are what allow later tokens to “look back” at the context that came before them.
Prefill is the first phase of inference. During prefill, the model processes the full input prompt — for example, the system prompt, user history, retrieved documents, and the latest question. All prompt tokens are known upfront, so the GPU can compute their K/V values in parallel. This makes prefill highly parallelizable, but it can still be expensive when prompts are long. It also directly affects time to first token, because the model cannot start generating the answer until the prompt has been processed.
Decode is the second phase. Once prefill is complete, the model starts generating the response one token at a time. Each new token uses attention over the cached K/V values from the prompt and from the tokens that have already been generated. After a token is produced, its own K/V values are added to the cache, and the process repeats for the next token. Unlike prefill, decode is inherently sequential: token 25 cannot be generated before token 24 exists. This is why decode is often slower from a throughput perspective and why generation speed is usually measured in tokens per second.
The important point is that the K/V cache is not optional. Without it, the model would have to recompute the entire context again and again for every generated token. The cache is what makes modern LLM serving practical.
Prefix cache extends this idea across requests. Many LLM requests start with the same prefix: the same system prompt, the same application instructions, the same chat history, the same tool definitions, or the same retrieved context. If request A already computed K/V values for that shared prefix, request B should not have to compute them again. Instead, the system can reuse the stored K/V values for the shared prefix, compute K/V only for the new suffix tokens, and then continue decoding normally.
This is especially valuable in real-world AI services. Chatbots, coding assistants, agents, and RAG applications often send long repeated prefixes with every request. Reusing the prefix cache reduces redundant GPU work, lowers time to first token, and improves overall GPU utilization. In simple terms: the model can spend less time rereading the same context and more time generating useful output.
The challenge is that K/V caches are large. They are not just a few metadata entries; they contain tensors for many tokens, across many layers and attention heads. Keeping all reusable prefixes in GPU memory is expensive and limited by HBM capacity. To make prefix reuse work at scale, cached K/V values need to live somewhere that can hold a large amount of data and feed it back quickly when the same prefix appears again.
That somewhere is storage.
Disaggregated serving makes the storage layer critical
The trend in inference serving architecture is disaggregation. NVIDIA's Dynamo framework is the clearest articulation of the pattern. In a traditional (coupled) serving setup, one GPU group handles both prefill and decode — a compromise, since prefill is compute-bound (heavy matrix-matrix work) while decode is memory-bandwidth-bound (frequent small data movement). In a disaggregated setup, separate GPU pools handle each phase, each configured for its own requirements. The KV tensors computed by the prefill pool are transferred over the network to the decode pool.
The consequence for storage is direct: the KV cache is no longer a local concern. It is shared state that moves across nodes through the storage and network fabric. Every miss pays full network latency. That is why disaggregated serving makes storage architecture even more important.
Inference storage is not one workload but four
One of the most important reframes in our work is recognizing that "AI inference storage" is not a single I/O pattern. It is at least four, each with different characteristics, and Lustre fits each one differently.
1. Data plane: model loading and staging
The first storage plane is the one Lustre was effectively built for. Loading a model means reading tens to hundreds of gigabytes per shard — and for a full model, you're in TB-class territory. The access pattern is almost entirely sequential: a heavy read burst on cold start, then silence until the next reload. There are no small random reads, no mixed workloads, no surprises.
For this plane, shared throughput storage with RDMA capability is the natural fit, and Lustre delivers it without compromise. OST striping across multiple targets combined with LNet over InfiniBand or RoCE aligns directly with the GPU fabric topology, and the production track record speaks for itself — Frontier, Aurora, and Leonardo all load their models this way. If inference stopped here, the storage problem would already be solved.
2. Context plane: retrieved knowledge and RAG
The second plane looks quite different. Retrieval-augmented generation works by querying a shared vector index at inference time — many concurrent requests, each fanning out across the index in reads of 4 to 48 KB. Once the index is built, more than 95% of accesses are read-only, but the I/O pattern is the opposite of the model loading plane: high fan-out, small, and random rather than large and sequential. On top of that, cached document-KV chunks come in at MB scale when a query hits a previously computed result.
Lustre's fit here is mixed, and it's worth being precise about why. The stock Lustre client was not designed for high-IOPS small random reads — that's not a criticism, it's just not what the filesystem was built to do. Data-on-MDT can help when the RAG/index working set is represented as small immutable files or DoM-sized shards created with the right layout policy. In that case, small reads can avoid OST RPCs and be served from NVMe-backed MDTs. However, DoM is not a general acceleration mechanism for random 4–48 KB reads inside large ANN index files. For large index objects, Lustre striping, client-side caching, local NVMe cache remain more relevant.
3. Control plane: session artifacts and traces
The third plane is the one we understand least, and we want to be upfront about that. The workload is write-dominant — many concurrent workers appending records continuously — with reads arriving in two modes: occasional bursts for debugging and auditing during a run, and heavier analytical access after the fact. What we don't yet have is a rigorous characterization of the size profile, because the research literature is only beginning to catch up with how agentic systems generate and consume trace data.
The schema is starting to crystallize — AgentTrace from UC Berkeley defines three trace surfaces the storage layer needs to absorb (operational, cognitive, and contextual), and Hindsight demonstrates retroactive sampling at nanosecond-level overhead and GB/s per node throughput — but a proper I/O characterization of this pattern on a parallel filesystem doesn't exist yet. If you're running agentic workloads at scale and logging session artifacts to Lustre, we'd genuinely like to hear from you.
4. Context memory: the KV cache
The fourth plane is where the most active storage research is happening, and where the fit question is most nuanced. The KV cache doesn't live in one place — it spans a hierarchy from GPU HBM down through CPU DRAM, local NVMe, and finally remote shared storage, with I/O sizes ranging from 64 KiB to several dozen MiB. The access pattern is mixed: sequential appends during prefill as the cache is written, and large block reads during decode as the model pulls cached attention states back in. Decode-phase bandwidth demand alone can span anywhere from 2 to 200 GiB/s depending on model size and context length.
Lustre's role here is specific rather than general. It is not the right home for the hot end of this hierarchy — HBM-resident data moves too fast and lives too briefly for a network filesystem to be in that path. But for the colder tiers, where KV state is offloaded to shared storage between requests or across sessions, Lustre is a strong candidate. That's the tier we focused on in our testing, and the rest of this post is about what we found there.
Part 2. A new storage layer for AI inference
Where Lustre sits in the inference hardware stack
NVIDIA's CMX/Dynamo architecture defines four hardware memory tiers for inference:
- G1 — GPU (Rubin) with HBM. Closest to the compute, fastest, smallest.
- G2 — CPU node (Vera) with DRAM. Larger, still fast.
- G3 — Local SSD on the same server.
- G3.5 — BlueField-4 storage backend with NVMe JBOFs, accessed over Spectrum-X Ethernet. This is the new disaggregated NVMe tier.
- G4 — Durable external storage. Holds inactive multi-turn KV state, query history, logs, and artifacts that may be recalled in future sessions.
Lustre fits at G3.5 and G4. These are exactly the layers a parallel filesystem can serve.
Source: NVIDIA CMX / Dynamo architecture, as shown in Blocks & Files, “Nvidia’s basic context memory extension infrastructure”
Where Lustre sits in the inference software stack
The software stack is fragmented. At the orchestration layer sit Triton Inference Server, NVIDIA Dynamo, and KV Cache block managers. Below that are the inference engines — vLLM, TensorRT-LLM, SGLang, ONNX, OpenVINO — each with its own KV cache logic. On the storage backend side: GPUDirect, Redis, S3, Infinistore, Mooncake, Valkey, WEKA.
LMCache sits in the middle, providing a uniform KV caching interface between inference engines and storage backends. NIXL (NVIDIA's new low-level I/O interface) sits below LMCache and standardizes how it talks to backends.
For Lustre to integrate cleanly, it needs to plug in at the LMCache or NIXL layer. That is what we did in our implementation.
Our implementation stack
Top to bottom:
- vLLM running the model on GPU, with the KV cache living alongside
- LMCache — KV caching engine, handling policies and routing
- Lustre client with RO PCC (read-only Persistent Client Cache, Lustre 2.17.52)
- Lustre MDS + OSS running on top of xiRAID Classic
- STX platform — serves as both MDT components and PCC caching devices, connected via NVMe-over-Fabrics
- xiRAID Opus on the storage targets — RAID groups using SPDK in user space, polled mode I/O (not interrupt-driven), with AVX2/ARM vectorized RAID parity calculation
The xiRAID Opus design is worth noting: by running entirely in user space with polled mode, it eliminates the latency overhead of kernel interrupt handling and keeps one or more CPU cores fully occupied for maximum throughput.
MLPerf testing
To isolate the cost of each layer, we built three configurations on identical hardware: same drives, same network topology, same 2x200 Gb/s NVMe-over-Fabrics links to the storage targets. The only thing that changes between them is the filesystem stack.
The first configuration, XFS, is the local filesystem baseline. MLPerf Storage runs directly on XFS, on top of xiRAID Classic, with no network filesystem in the path — this is as close to bare metal as the benchmark gets. The second, Lustre, puts a Lustre client and 8 OSS nodes between the benchmark and the storage, while keeping everything else identical. This is where we measure the raw cost of routing through a parallel filesystem. The third, Lustre with RO PCC, adds read-only Persistent Client Cache on the Lustre client. Everything below it stays the same.
The structure is deliberate: XFS gives us the ceiling, Lustre shows us what the parallel filesystem costs, and Lustre with PCC shows us how much of that cost the cache recovers.
Test matrix dimensions:
Model (KV arch): llama3.1-70b-instruct dense GQA; gpt-oss-120b MoE; deepseek-v3 - MLA
Workload shape: synthetic-chatbot; synthetic-coding; synthetic-document; sharegpt; burstgpt
https://github.com/mlcommons/
- Scenario 1 — caches ON, capacity-mode autoscaler (min=10 max=500)
- Scenario 2— caches OFF --cpu-mem-gb 0 --disable-multi-turn --disable-prefix-caching
Phase: prefill-only; decode-only; "cold" decode in the PCC variant/warm decode in the PCC variant
Fixed benchmark parameters:
Duration: 180s per experiment (XFS/Lustre);
--max-concurrent-allocs 32
--storage-capacity-gb 200
Trials: 1 per experiment
Experiments counts and skipped cells:
XFS: 55 experiments — 5x llama3.1-70b s1 decode skipped
Lustre: 40 experiments — all llama3.1-70b skipped, DIO kernel panic under investigation
Lustre + RO PCC: 48 experiments — 70B excluded (same panic), BurstGPT excluded (trace-rate-bound)
Limitations
We want to be explicit about four constraints before presenting numbers:
- Pre-release software. MLPerf Storage 3.0 is still under development. Bug-fix patches and tuning are still landing; the same workload may behave differently in the GA release.
- Pending patch. A patch we applied gave ~15% better throughput in our testing. It has not yet been accepted upstream. Some of our measured headroom is contingent on that change.
- Internal testing. These results are from a single in-house testbed. They have not been submitted to any external organization or independently reviewed. Reproducibility on a different cluster is not yet established.
- Scope. Only the most signal-bearing cells from the full sweep are shown.
I/O characterization on XFS: what the workload actually looks like
Before comparing filesystems, it is worth understanding what the KV-cache benchmark actually generates on disk. These are XFS measurements — the baseline — showing real I/O at the block layer.
Scenario 2 prefill (writes only — reads are near-zero):
Scenario 2 decode (reads dominate, writes are fsync flushes):
Several observations from this data:
- Prefill is write-only and sequential. Average write sizes of 140–328 MB with peaks at ~1 GB. Average sustained bandwidth of 8.7–9.5 GB/s with peaks up to 25 GB/s.
- Decode is strongly read-dominant. Read sizes average 11–18 MB per I/O, peak at 23 MB. Read bandwidth peaks at 13–27 GB/s. Writes during decode are fsync flushes — background, not in the critical path.
- Queue depth is bursty. Peaks of 44–75. A parallel filesystem with multiple OSTs and concurrent I/O paths handles this well.
This is not a difficult workload for a parallel filesystem in principle. The question is whether the overhead of going through Lustre erases the advantage.
Throughput results
All figures below are normalised to XFS = 100%. Higher is better.
Summary (averaged across all models and workloads):
Three takeaways:
- Lustre alone trails XFS by 15–20%. This is the cost of routing through a parallel filesystem stack instead of local XFS over NVMe-oF.
- PCC closes most of the decode gap, and then some. Decode average tokens/s at 104% of XFS — a net win over the local filesystem baseline.
- PCC has minimal effect on prefill. PCC is a read cache. Prefill writes the cache; there is nothing yet to serve from it, so PCC d1 and d2 converge near each other.
Detail breakdown — prefill (avg tokens/s vs XFS):
The per-workload decode table tells the real story. DeepSeek V3 synthetic-chatbot scenario 2 decode is the most striking: Lustre alone sits at 57% of XFS, while PCC d1 reaches 107% and PCC d2 reaches 108% — a 50-point swing from a single cache layer. DeepSeek V3 synthetic-coding scenario 2 reaches 114% on both PCC runs. GPT-OSS-120B ShareGPT scenario 1 hits 122% on the first decode run. These are not edge cases; they are a consistent pattern across decode workloads once the PCC cache has had time to warm.
The prefill table shows smaller and less consistent gains, which is mechanically expected: PCC is a read cache and prefill is primarily writing into it.
Latency results
All figures below are normalised to XFS = 100%. Lower is better.
Summary (averaged across all models and workloads):
Prefill latency is essentially unchanged. Both Lustre and Lustre+PCC sit around 109–113% of XFS on both mean and p95. This is a small, operationally acceptable regression.
Decode mean latency: Lustre is 1.4x XFS; PCC brings it to 1.05x. The overhead of going through the Lustre network stack shows up clearly without caching. PCC eliminates most of it.
Decode p95 (tail) latency: Lustre is 1.46x XFS; PCC brings it to 1.01%. This is the number that matters most in a production inference system. Tail latency on decode is what users feel as slow first-token response. PCC closes the tail gap almost completely, landing within 1% of the local filesystem baseline.
Taken together with the throughput results, the picture is consistent. Without PCC, routing through a parallel filesystem carries a real cost — roughly 15–20% on throughput and 1.4x on decode latency — but with PCC that cost largely disappears, decode throughput matches or beats XFS on most workloads, and decode tail latency falls within 1% of it. Lustre with RO PCC is not a compromise: on the metrics that matter most for inference, it is a genuine competitor to local storage, and in several workload cells it is the outright winner.
Eviction policy study
Lustre 2.17.52 ships PCC with a fixed eviction path. The natural follow-up question after seeing the PCC results is: is that fixed choice the best one? Could a different eviction policy do better?
To answer this, we designed an offline simulation study.
Methodology
Step 1 — Capture.
We collected real KV-cache I/O traces from three models — llama3.1-8b, gpt-oss-120b, and deepseek-v3 — running both with PCC off (phase 1) and PCC RO autoattach (phase 2). Each capture ran for 20 minutes per (model, phase) combination, instrumented with strace -y, bpftrace probes on the PCC kernel symbols, and llite stats.
Total: 30 traces, 8.0M I/O events, 281GB working set, 6 cache frictions, 1,260 runs.
Step 2 — Replay.
All 30 traces were replayed through 7 eviction policies in an offline simulator, across 6 cache size fractions:
belady: oracle MIN — upper bound with perfect future knowledge
lru: byte-LRU baseline (what PCC currently approximates)
2q: Kin/Kout — promote on 2nd hit
arc: adaptive recency/frequency balance
clock_pro: hot/cold/ghost CLOCK
lfu_aging: frequency + counter halving
tinylfu_slru: CMS admission door + SLRU
Total: 1,260 simulator runs
Step 3 — Score.
Each run was scored on three metrics:
- hit_ratio = hits / total_ops: How often a request is served from cache. Higher = lower latency.
- amplification = backend_bytes / hit_bytes: Bytes pulled from the backend per byte served from cache. Lower = less wasted I/O.
- wasted_ratio = admits never reused: Did this admission decision matter? Lower = fewer pointless inserts.
- The Pareto frontier of hit_ratio ↔ amplification is what reveals the regime split.
The two-regime finding
Plotting mean hit_ratio against mean amplification across all 30 traces and 6 cache fractions reveals two distinct operating bands:
Latency-bound cluster: ARC, LRU, and 2Q sit nearly on top of each other. They form the Pareto front for hit ratio. CLOCK-Pro falls just behind them. These policies are within ~1 percentage point of each other on hit_ratio — the difference between them is negligible in terms of measured output, and the choice is dominated by implementation complexity, not numbers. LRU wins by simplicity.
Bandwidth-bound cluster: TinyLFU+SLRU sits alone, roughly 33 percentage points below the top cluster on hit_ratio, but with amplification of ~0.035 — approximately 10x lower than LRU. Its CMS admission door filters out single-use data before it enters the cache, dramatically reducing backend pressure at the cost of hit rate.
For the workloads in our throughput sweep — which are latency-bound — the current PCC approximation of LRU is already near-optimal. The case for TinyLFU is real, but it applies to deployments where the OSS or OST bandwidth is the binding constraint.
Tuning sweep: does parameter choice matter?
We ran a second sweep to rule out the possibility that the underperformers' results were artefacts of default parameters: 13 variants x 30 traces x 6 cache fractions = 2,340 additional simulator runs.
For TinyLFU + SLRU parameter sweep à Spread: 0.423 → 0.438 — 1.5 pt across all five variants. The only useful tweak is setting cms_width = 65,536 , which gives +0.8 pt hit_ratio at near-zero memory cost. The prob_frac parameter is a structural no-op: the SLRU promotion door fires so infrequently in this workload that its setting does not matter.
For LFU-aging parameter sweep à Spread: 0.544 → 0.561 — 1.7 pt; age_shift has zero effect.
Verdict: age_shift had literally zero effect. In a 320,000-operation trace with age_every = 10,000 , the halving fires only ~30 times — so rare that the magnitude of the cut does not matter. No combination of parameters closes the gap to LRU. The gap is structural: LFU-aging has no recency component, and no tuning can supply one.
What comes next
Our current results are on a single testbed with headroom remaining in the storage network. We have a three-stage plan to build toward production-shaped validation.
Stage 1. Vertical scaling and testbed optimization. Drive the 400 Gb storage network to full utilization, at least on selected tests, while keeping latency below threshold. Key question: how does RO-PCC behave when the storage network is no longer underutilized? Today we have network headroom. Stage 1 removes it.
Stage 2. Scale-out configuration. Add more storage resources, OSS nodes, and Lustreintegrated clients. Lustre's native load distribution should provide additional parallelism, but we want to measure rather than assume. Key question: does RO-PCC improve performance in a horizontally scaled Lustre configuration?
Stage 3. Real workload validation. Move from synthetic KV-cache benchmarks to an agentic closed-loop workflow on a real, complex software project. This exercises model loading, RAG retrieval, KV cache, and session artifact storage simultaneously, under realistic workload timing. Storage beyond KV cache — source code, documentation, tool outputs — enters the picture. Key question: which Lustre settings actually improve real agentic workload behaviour?
Our results show that Lustre with RO-PCC is already competitive with local XFS for KVcache inference workloads:
- End-to-end decode throughput at 104% of XFS on average, with cells reaching 114–122% on specific workloads
- Decode p95 tail latency within 1% of XFS
- Prefill with no meaningful regression
Without PCC, the cost of going through a parallel filesystem stack is real — roughly 15–20% throughput and 1.4x decode latency. With PCC, that cost largely disappears. The cache warms fast, and when it is warm, Lustre+PCC is a net win on the most latency-sensitive metric inference: the decode tail.
The eviction policy study adds one actionable result: for latency-bound serving, LRU is already near-optimal and PCC's current fixed policy is close enough. For bandwidth-bound serving, TinyLFU+SLRU appears to be the better candidate for bandwidth-bound deployments, where reducing backend byte amplification matters more than maximizing operation-level hit ratio.
