Why Gradient Descent Became Stochastic

The mathematical foundation of simple linear regression is explored by deriving the formulas for its slope and intercept parameters. This derivation begins with the Mean Squared Error (MSE) loss function, which is visualized as a 3D bowl curve where the goal is to find the minimum point. Using calculus, partial derivatives of the MSE function are taken with respect to both the slope and intercept. By setting these derivatives to zero, a system of equations is solved to produce the closed-form formulas for the optimal parameters. This analytical solution, however, becomes computationally complex for datasets with many features, motivating the need for iterative optimization algorithms like gradient descent.