FairStress: A deeper evaluation of fairness in LLM decision-making

FairStress is a new benchmark designed to measure demographic fairness in LLM decision-making beyond just the final output. It evaluates how readily a model changes its decision when challenged with persuasive arguments, known as "adversarial prefixes," depending on the demographic groups involved. Across various models, including the GPT and Llama families, a pattern of "anti-stereotypical over-correction" was observed. This means models were more willing to abandon a correct, evidence-based decision when doing so favored a minority-coded candidate. This benchmark reveals hidden asymmetries in how consistently models follow evidence across different demographic scenarios in high-stakes domains like hiring and lending.