olmo-eval: An evaluation workbench for the model development loop

olmo-eval is an evaluation workbench designed to support the iterative development loop of large language models. It extends the OLMES standard to provide more flexibility for implementing new evaluations and composing them into larger workflows. The tool differs from existing frameworks by separating benchmark logic from runtime policy, supporting agentic evaluations in sandboxed environments, and offering stronger analysis tools to assess model improvements. Its modular design allows for swappable components like models, tools, and containerized environments, making it easier to adapt evaluations as a model evolves.