How to Analyze and Optimize Your LLMs in 3 Steps

A three-step process is outlined for analyzing and optimizing Large Language Models (LLMs) that are already in production. The first step involves analyzing LLM outputs through manual inspection, grouping user queries into a taxonomy, and using an LLM-as-a-judge approach on a golden dataset. Next, improvements are made iteratively by fixing obvious issues, using prompt optimizers, and prioritizing tasks with a value-effort graph. The final step is to evaluate the changes against quantitative metrics and iterate on the process until the model's performance is deemed sufficient.