Why Parameter Count Is Not Model Complexity: An Introduction to Effective Dimension

The number of parameters in a machine learning model is often a misleading indicator of its true complexity due to parameter redundancy. A more nuanced metric, known as effective dimension, quantifies the number of genuinely independent ways a model can influence its output. This measure provides deeper insights into a model's capacity, generalization performance, and trainability beyond what loss or accuracy can reveal. The core mathematical tool for this analysis is the Fisher information matrix, which measures how sensitive a model's predictions are to changes in its parameters.