Beyond Prompt Caching: 5 More Things You Should Cache in RAG Pipelines

Caching can be applied to multiple stages of a Retrieval-Augmented Generation (RAG) pipeline to improve efficiency beyond just prompt caching. The two main types are exact-match caching, which uses key-value stores for identical queries, and semantic caching, which uses vector databases to find similar queries. Specific caching layers include the query embedding cache, which stores embeddings for repeated or normalized queries to avoid re-computation. Additionally, a retrieval cache can store the document chunks retrieved for a specific query, and a reranking cache can store the reordered results from a reranker model. These techniques help reduce latency and cost in high-traffic AI applications by reusing previously computed results.