Measuring the Creativity Potential of LLM Agents

The creativity of LLM agents is evaluated to determine their potential for scientific discovery. Creativity is defined by originality and usefulness, which are further broken down into P-creativity (personal novelty), H-creativity (historical novelty), and impact. A framework is proposed to measure these metrics on ML engineering tasks, using an LLM-as-a-judge approach to score solutions against prior agent attempts and a large corpus of human solutions from Kaggle. The study finds that agents tend to shift from exploration to exploitation over time, with creativity declining as they refine their solutions, which helps explain performance differences between agent frameworks.