Does Stacking Beat Your Best Model?

Stacking, an ensemble machine learning technique, is most effective when its base models make different types of errors. An experiment demonstrates this by comparing a stack of diverse models (linear, tree, SVM) against a stack of correlated models (three tree-based ensembles) on a synthetic dataset. The diverse stack showed a meaningful accuracy improvement over the best single model, whereas the correlated stack offered a negligible gain at a much higher computational cost. The analysis also finds that simple voting can underperform the best single model because, unlike stacking's meta-model, it cannot learn to weight more reliable models more heavily.