Large language models forget expensive work whenever their temporary context is discarded. A new reference architecture from Supermicro, IBM and NVIDIA treats that intermediate state as shared storage, moving reusable key-value cache data beyond scarce GPU memory and onto a networked flash tier.
In a 5 November 2026 announcement, the companies reported up to a 56-fold reduction in time to first token for a 130,000-token prompt compared with recomputing the same context. That is a vendor benchmark on one defined stack, not a universal speed-up for every model or application.
KV cache is the model’s working memory
Inference has two broad stages. During prefill, a model processes the input and creates key and value tensors that represent relationships among tokens. During decode, it reuses those tensors while generating the response one token at a time. Keeping the KV cache avoids repeating the expensive prefill when a later request shares the same context.
Long documents, coding repositories, retrieval systems and multi-step agents can create far more cache data than fits in GPU high-bandwidth memory. CPU memory, local drives and remote storage form progressively larger but slower tiers. The engineering question is whether moving a cached block is faster than calculating it again.
The design combines storage, networking and an inference scheduler
The tested platform used eight Supermicro Petascale storage nodes running IBM Storage Scale, NVIDIA Spectrum-X networking and NVIDIA Dynamo to manage inference and cache placement. Five client nodes accessed the shared storage, while a quantized 120-billion-parameter model ran through a vLLM-compatible endpoint.
NVIDIA’s Dynamo architecture can route requests toward workers that already hold useful cache blocks and offload blocks across GPU, CPU and storage tiers. That turns caching from a feature inside one server into a distributed scheduling problem.
The benchmark rewards reuse, not every request
The headline comparison measured a warm persistent cache against cold execution that recomputed a 130,000-token prompt. In that favourable reuse case, time to first token improved by up to 56 times. The accompanying IBM reference paper also reports up to 22-fold higher request throughput under concurrent load and an 18-fold advantage during a noisy-neighbour network test.
Those numbers are meaningful for repeated context, such as an agent revisiting the same codebase. They say less about one-off prompts, frequently changing context or applications dominated by token generation rather than prefill. Cache misses still require computation, and remote reads add network latency.
Shared context creates operational questions
A persistent cache can improve GPU utilization and let storage capacity scale separately from compute. It can also support failover because useful state is not trapped inside one GPU. But operators must decide what to retain, when to evict it and how to prevent one workload from crowding out another.
Security becomes more important as well. Cached context may contain proprietary code, personal information or retrieved documents. Isolation, encryption, access control and deletion policies need to follow the data beyond the original inference worker. Faster reuse is not worth accidental cross-tenant exposure.
Inference infrastructure is becoming a memory hierarchy
The reference design illustrates a broader shift from buying only more accelerators to organizing memory and data movement around them. For agentic and long-context workloads, a storage system can influence response latency almost as directly as the GPU.
Independent tests across models, prompt patterns, cache-hit rates and competing storage systems are still needed. The result is a credible engineering demonstration with unusually large gains in a reuse-heavy benchmark—not proof that shared flash makes every AI service 56 times faster.
Featured image: Server racks in a data centre. Contextual image, not the tested Supermicro–IBM–NVIDIA system. Photo: Tyler/Unsplash.
Leave a Reply