How to Estimate the Recall of AI Agents with Capture-Recapture

The capture-recapture statistical method, traditionally used in ecology to estimate animal populations, can be adapted to evaluate the recall of AI agents. By comparing the outputs of two independent AI models or processes, this technique estimates the total number of items, including those missed by both agents. An experiment on the CUAD legal contract dataset demonstrates that this method provides a more accurate measure of performance than relying on a single model for evaluation. The approach is particularly useful for identifying 'unknown unknowns,' as models from the same family often share blind spots, leading to overconfidence in their performance.