Stop Using LLMs Like Giant Problem Solvers

An attempt to transform 100 messy compliance PDFs into structured JSON rules using a single large LLM agent proved unreliable and difficult to scale. A more successful approach involved changing the shape of the problem by making the agent's job smaller and more focused. This was achieved by pre-processing data, processing documents iteratively, and using deterministic code for tasks like parallelization, validation, and caching. The system became more reliable by making outputs traceable and recoverable, highlighting the need for a hybrid approach where LLMs handle semantic judgment and code handles structure and control.