Building Fair Evaluation Sets Is a Combinatorial Problem

High overall model accuracy can dangerously mask poor performance on minority groups, as imbalanced evaluation datasets absorb major failures into a single healthy-looking number. Creating a fair evaluation set is a complex combinatorial problem because it requires balancing multiple attributes like age, race, and gender simultaneously. This challenge can be solved by framing it as an integer programming problem, which mathematically finds the optimal subset of data that best matches target distributions for all attributes. An open-source tool called `datacarve` implements this method, enabling teams to "carve" out a balanced evaluation set where every group has equal statistical footing for more reliable measurement.