I Measured Neural Network Training Every 5 Steps for 10,000 Iterations

Contrary to common belief, neural networks dynamically expand their representational capacity during training rather than simply exploring a fixed, predetermined space. High-frequency monitoring reveals this expansion occurs in distinct phases, with most capacity-building "jumps" happening surprisingly early in the training process. This insight suggests that feature formation is a sequential process, meaning interpretability analysis should focus on these early stages, not just the final model. Counterintuitively, as a network's performance improves and its loss decreases, its internal complexity actually increases to build the necessary structures for the task.