Zero-Waste Agentic RAG: Designing Caching Architectures to Minimize Latency and LLM Costs at Scale

Enterprise Retrieval-Augmented Generation (RAG) systems suffer from high costs and latency due to redundant, semantically similar user queries. A two-tier caching architecture is proposed to mitigate this, featuring a semantic cache for identical queries and a retrieval cache for shared context. The semantic cache returns pre-generated answers for highly similar questions, while the retrieval cache provides pre-fetched data to the LLM for new answers on similar topics, skipping expensive database lookups. An intelligent agent orchestrates this process, using a toolkit of functions to validate cache staleness and route queries effectively between the cache and live data sources to reduce costs and latency.