How Clay uses LangSmith to debug, evaluate, and monitor 300 million agents runs per month

Clay, a platform for go-to-market teams, runs approximately 300 million AI agent runs per month for tasks like lead sourcing, AI-powered research, and drafting personalized outreach. This massive scale created challenges with quality control, unpredictable costs across multiple model providers, and the rapid pace of new model releases. To solve these problems, Clay adopted LangSmith as its observability and evaluation backbone for its custom agent framework. LangSmith provides detailed trace visibility for debugging, production monitoring for costs and errors, and a structured evaluation system for comparing models. This has allowed Clay to achieve near-perfect cost reconciliation, compress debugging loops, and make informed decisions about model routing and pricing.