tierkv added to PyPI
News Source : Pypi.org
News Summary
- 3-tier distributed KV cache for LLM inference — preserve evicted KV across cluster nodes.
- ExO integration (BF16, 8k–15k token prompts):vLLM integration (Apple FY2025 10-K, GB10 GPU, real-world document Q&A):Cold vault restore beats GPU cache hit — blocks land directly into the KVcache skipping attention recomputation entirely.
- The speedup grows with context length because prefill scales super-linearly while restore is near-linear (network transfer).
- Answer quality is bit-for-bit identical across all three paths.
3tier distributed KV cache for LLM inference.When your GPU evicts a KV cache entry, tierkv ships it to another machine over gRPC instead of dropping it. On the next request with the same prompt, t [+18013 chars]
Never miss a story from us, subscribe to our newsletter