Introducing Align Evals: Streamlining LLM Application Evaluation

Align Evals is a new feature in LangSmith designed to calibrate LLM evaluators to better match human preferences. It addresses the challenge of evaluation scores not aligning with human expectations, leading to noisy comparisons and wasted time. The feature provides a playground-like interface for iterating on evaluator prompts, side-by-side comparison of human-graded data and LLM-generated scores, and a saved baseline alignment score for comparing changes. The alignment flow involves selecting evaluation criteria, selecting data for human review, grading the data, and creating/testing an evaluator prompt against human grading.