What the ReLU Revolution Revealed About Biological Plausibility

The shift from sigmoid to Rectified Linear Unit (ReLU) activation functions was driven by the practical need to train deeper neural networks. Sigmoid functions, despite their perceived biological plausibility, suffered from the vanishing gradient problem, where their small derivative prevented effective learning in deep architectures. ReLU solved this by having a derivative of one for positive inputs, allowing gradients to propagate effectively and dramatically improving training speed. Ironically, while often viewed as a pragmatic hack, ReLU's sparse, threshold-based activation was argued to be more biologically plausible than the sigmoid's. This history demonstrates that empirical performance, rather than strict biological mimicry, became the primary driver for architectural choices in deep learning.