A Neural Network Learned to Read Digits From Pure Noise

A student neural network can learn to classify handwritten digits with high accuracy despite only being trained on random noise. This is achieved through a process where the student network starts with the same initial weights as a teacher network that was trained on the actual digits. The student's sole training task is to mimic a few auxiliary outputs from the teacher when both are fed the same random noise. This knowledge transfer is highly effective, but the experiment demonstrates it fails completely if the student and teacher do not share the same starting weights. The shared initialization provides a common coordinate system that allows the student's weight updates to align with the teacher's learned changes.