Copilot Caught 8 of 8 Diff Bugs With Rules, 0 of 2 Outside the Diff

An experiment tested GitHub Copilot's code review capabilities by seeding ten bugs into pull requests against a synthetic billing service. When provided with custom rules, Copilot caught all eight bugs located within the changed lines of code, but missed the two bugs that sat in unchanged, dependent code. Without rules, it caught only five of the eight bugs within the changed code and still missed the two outside of it. This highlights a "context gap" where AI reviewers are effective at spotting local mistakes but can miss systemic issues one call away from the changed code. The results suggest that a clean review from an AI agent does not guarantee a change is safe for the wider system, and human oversight remains critical.