Agentic AI: How to Save on Tokens

Agentic AI systems can become very expensive in production due to the large number of tokens required for system prompts, tools, and conversation history. Several design principles can be implemented to mitigate these high costs and save on tokens. A primary technique is prompt caching, which avoids re-processing static parts of a prompt by storing and reusing their K/V tensor computations, leading to significant cost reductions on subsequent calls. This is implemented as prefix caching in frameworks like vLLM and is supported with specific rules by API providers like OpenAI and Anthropic, while semantic caching offers another method by matching requests based on meaning.