Inference & performance · Concept

KV cache

Stored attention keys and values reused to avoid recomputing prior-token attention state.