A Soft-Label MNIST Testbed for Human vs. Model Confidence

A new dataset is introduced to explore model uncertainty by re-labeling the MNIST and Mukhoti digit datasets with human annotations. Annotators marked each image with "yes," "no," or "unsure" for every possible digit, generating soft-labels that capture ambiguity and disagreement. This process provides a more nuanced ground truth than traditional one-hot labels, especially for images where humans are uncertain. The resulting testbed is designed for studying how machine learning models handle ambiguous data compared to human perception. The dataset and associated uncertainty metrics are available on Hugging Face.