Stop Blaming the Data: A Better Way to Handle Covariance Shift

Covariance shift occurs when a model's performance drops because the feature distribution in new data differs from the training data. Instead of using this as an excuse, Inverse Probability Weighting (IPW) offers a solution by re-weighting the validation data to mimic the new data's distribution. This is practically achieved by training a binary classifier, or a propensity model, to distinguish between the validation and test datasets, using its predictions to derive the necessary weights. By applying these weights during evaluation, one can create a synthetic cohort that matches the test domain, enabling a more accurate and fair assessment of model performance.