My Model Was Cheating on Its Own Test

Data leakage can occur when a model inadvertently sees information from the test set, leading to overly optimistic performance metrics. A car price prediction model initially reported a high R-squared score of 0.887 due to a flaw in its preprocessing pipeline. Preprocessing steps like scaling and outlier handling were applied to the entire dataset before it was split into training and testing sets, allowing the model to "peek" at test data. After correcting the pipeline to split the data first and fit preprocessors only on the training data, the true R-squared score was revealed to be a much lower 0.767. This case study highlights how a simple mistake in the order of operations can cause a significant, yet easy-to-miss, data leak.