Sequential Fitting: A Different Perspective on the Spectral Bias of Neural Networks

Neural networks exhibit a "spectral bias," meaning they learn low-frequency components of a function before high-frequency ones, which slows down training for complex, oscillatory functions. An alternative perspective called "sequential fitting" is proposed, where networks fit a target function starting from the domain boundaries and progressing inward. This process involves building up the fit one oscillation at a time, as demonstrated through one-dimensional regression examples using MLP networks. The training is also influenced by a "boundary effect," where the function's behavior near the edges of the domain impacts how the sequential fitting proceeds. This behavior is interpreted by examining the step function-like basis that the network iteratively learns in its final layer.