Why Care About Prompt Caching in LLMs?

Prompt caching is a technique used to optimize Large Language Model (LLM) calls by reducing latency and cost. It works by storing the computed state of common prompt prefixes, such as system instructions or retrieved context, after the first request. When a new request shares the same initial tokens, the model can reuse these cached computations instead of processing them from scratch. This method extends the concept of KV caching from single-prompt inference to work across multiple user sessions, proving especially useful for applications with repeated instructions like RAG pipelines.