I Built a Trap for AI Image Judges. They Walked Into a Different One.

An experiment tested three vision AI models (GPT-4o, Claude, Gemini) on their ability to judge image quality based on specific criteria. The test used 60 image pairs, including 'trap' pairs, to see if models focused on the requested detail or were swayed by overall aesthetics or image order. The results revealed significant flaws, such as Claude demonstrating a strong positional bias by consistently choosing the second image. GPT-4o was found to invent answers when asked about differences that did not exist. The author also provides a Python script for others to conduct a similar, smaller-scale test on their own images.