We Built a Routing Layer to Cut Our AI Costs. It Broke the Product.

To reduce AI expenses, many teams are building routing layers that send simple queries to cheap models and complex ones to more capable models. One team successfully used this approach to slash their AI bill by more than half, with all initial metrics appearing positive. However, over the next three months, customer satisfaction dropped and churn increased as the cheaper model failed on nuanced queries that were misclassified as simple. This critical quality degradation went unnoticed because the team's monitoring systems aggregated data from both models, masking the poor performance of the cheaper tier. This scenario reveals a common "Pareto trap" where cost savings are structurally tied to an unseen loss in product quality.