The Hidden Cost of LLM Routing: Why Switching Models Costs More

Switching between different large language models to save money, a practice known as routing, often carries a significant hidden cost that can erase the intended savings. This expense occurs because a new model has no memory of the prior conversation and must re-process the entire history from scratch, an operation that invalidates the efficiency of prefix caching. Experiments reveal that frequent, per-request switching can destroy nearly twice as much cache as sticking with one model, drastically reducing expected savings. A more effective strategy is "harness routing," which makes decisions from within the agent's runtime to account for the task's context and minimize costly, cache-breaking model switches.