What is kv cache

Overview

KV cache, also known as Key-Value cache, is a critical optimization technique used in transformer-based language models during the inference phase. When a language model generates text token by token, the KV cache stores previously computed key and value vectors from the attention mechanism, eliminating the need to recalculate these values for earlier tokens in each new inference step.

How KV Cache Works

In transformer models, the attention mechanism computes queries, keys, and values for each token. Without KV caching, generating a long sequence requires recomputing keys and values for all previous tokens repeatedly, creating redundant calculations. With KV caching, these computed values are stored in memory. When generating the next token, only the new token's query is computed, while previous key and value vectors are retrieved from cache, dramatically reducing computation.

Performance Benefits

KV caching dramatically improves inference speed, particularly for longer sequences. For example, generating a 100-token sequence with caching requires substantially less computation than without caching because each new token only needs attention calculated against cached values rather than all tokens in the sequence. This speedup is especially important for real-time applications like chatbots where users expect low-latency responses.

Memory Tradeoffs

While KV caching significantly reduces computation, it increases memory requirements. Each token's key and value vectors must be stored for the entire sequence length. For large models generating long sequences, memory becomes the limiting factor. Batch processing multiple requests compounds memory pressure. Techniques like KV quantization and sliding window attention help mitigate memory costs while maintaining performance benefits.

Implementation and Optimization

Modern inference frameworks like vLLM, TensorRT, and others provide optimized KV cache implementations. Quantization techniques reduce numerical precision of cached values, decreasing memory usage. Techniques like paged attention (organizing KV cache as virtual pages) improve memory efficiency. Some systems use dynamic KV cache allocation, only storing necessary values based on attention patterns.