Measuring What Matters with NeMo Agent Toolkit

Observability and evaluation are crucial for production LLM applications, and the NeMo Agent Toolkit (NAT) provides features for both. The toolkit integrates with observability tools like Phoenix to trace application internals, such as tool calls, timings, and token usage, which helps identify bottlenecks and inefficiencies. For evaluation, NAT allows users to define datasets and use frameworks like Ragas to measure metrics including Answer Accuracy and Response Groundedness. The process involves updating a YAML configuration to specify evaluators and datasets, then running a command to generate detailed reports containing scores and execution traces for each test case.