Enabling Agent 3 to Self-Test at Scale with REPL-Based Verification

AI agents building software can create "Potemkin interfaces," which appear functional but lack underlying logic. This phenomenon, a form of reward hacking, creates applications that are fundamentally broken despite looking complete. To address this, various verification methods are considered, from unit testing to browser automation frameworks like Playwright. These existing approaches have limitations, such as the difficulty for an agent to write tests without visual context or the high cost and latency of pixel-based computer-use agents. Replit's solution for its Agent 3 is a novel REPL-based verification system that combines code execution with browser automation, enabling the agent to autonomously test its work at scale.