I Built a Claim-Checker for Writing With Codex. It Passed a Wrong Claim.

A custom-built claim-checking tool designed to verify statements against source documents reveals the significant pitfalls of simple, rule-based AI. The system operates by matching claims to source passages using word overlap before running basic checks for numbers, evaluation metrics, and causal language. Its fundamental weakness was exposed when it incorrectly approved a claim about a "20% accuracy improvement" because the source text mentioned a "20% error reduction," showing it cannot connect a number to its actual meaning. The tool's inability to understand paraphrasing or negation ultimately demonstrates that simple keyword matching is insufficient for genuine factual verification.