My Model Was Cheating on Its Own Test
| Source: Towards Data Science
Tags: data leakage, scikit-learn, MLPRegressor, model evaluation, preprocessing
A practitioner traced a 12-point R² inflation — from 0.767 to 0.887 — to two preprocessing lines running in the wrong order, letting a scikit-learn car price model see the test set before evaluation. The walkthrough makes data leakage concrete for anyone using sklearn pipelines.
Details
Data leakage is one of the most common silent killers of model validity, and this Towards Data Science piece makes the failure mode tangible. The author trained an MLPRegressor on UCI's Automobile dataset and initially reported a test R² of 0.887 — a strong result for under 200 cars. After auditing the pipeline, a preprocessing ordering mistake was found: scaling was applied before the train-test split, letting the scaler absorb information from the test set. Fixing the order dropped the honest test R² to 0.767 — still useful, but 12 points lower. The piece connects the personal example to academic literature, citing Kaufman et al.'s 2012 ACM paper on leakage detection, and the 2010 INFORMS stock prediction competition where competitors reverse-engineered hidden test identities from public data. The mechanics are clear: fit the preprocessor on training data only, then transform the test set using those fitted parameters. For practitioners, the takeaway is operational — sklearn's Pipeline class exists precisely to prevent this class of mistake by bundling preprocessing steps into cross-validation folds. The article serves as a useful debugging reference for teams reviewing legacy notebooks or onboarding junior engineers.