The Machine Learning “Advent Calendar” Bonus 2: Gradient Descent Variants in Excel

Gradient descent is a core principle for training weight-based models, but the basic version is often slow or unstable. This content explores several advanced variants designed to improve optimization speed, stability, and convergence. It covers momentum-based methods like Standard and Nesterov Momentum, which add inertia to accelerate progress in consistent directions. Additionally, it details adaptive gradient methods such as RMSProp and Adam, which adjust the learning rate based on the history of gradients to improve stability. All these concepts are illustrated using a simple function in Excel to visually compare how each algorithm moves toward a minimum.