How Podium optimized agent behavior and reduced engineering intervention by 90% with LangSmith

Podium, a communication platform for small businesses, utilized LangSmith to optimize its AI Employee agent. By establishing a rigorous testing lifecycle, they curated datasets and fine-tuned models to solve specific issues, such as the agent's inability to end conversations naturally. This process, which included model distillation and pairwise evaluations, improved the agent's F1 response quality to over 98%. Furthermore, by providing their support team with access to LangSmith traces for troubleshooting, Podium reduced the need for engineering intervention by 90%, allowing engineers to focus on development.