Loading
Life sciences · Preprint
arXiv · September 10, 2026
Early or partial results. Treat as a signal, not a conclusion.
This preprint describes py-kvcache, a systems connector for external key-value cache management in vLLM that uses asynchronous I/O and scheduler-aware preloading to reduce time-to-first-token (TTFT) on long-context requests. The tool achieves 2.0x faster loading from disk than LMCache at 80k tokens but provides no benefit on high-end GPUs (H100) when GPU memory is sufficient, suggesting the utility is hardware and workload dependent.
Systems performance characterization using synthetic workloads, long-context benchmarks, and production trace replays. Workloads ranging from long-context requests (80k tokens) to irregular prefix chains and multi-turn conversational scenarios; tested on both high-end (H100) and weaker GPU configurations. Intervention: py-kvcache: a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading. Compared with: LMCache and native vLLM KV Offload implementation.
At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x speedup With GPU, CPU, and disk caching enabled together, py-kvcache is 1.23x faster than LMCache and within approximately 4% of native vLLM KV Offload On H100 GPU with Bailian trace replay, average request falls below break-even point and GPU memory alone retains sufficient prefixes
Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.
The source did not state who this applies to in practice.
This is a systems performance characterization study of an experimental software implementation (py-kvcache) for optimizing LLM inference, presented as a preprint without peer review; it demonstrates engineering improvements in a specific tool but lacks clinical or health outcomes and is not yet published in a peer-reviewed venue.
As stated by the source record.
Quoted from the source exactly as published.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.