Your Synthetic Data Passed Every Test and Still Broke Your Model

Standard evaluation frameworks for synthetic data, focusing on fidelity, utility, and privacy, are often misleading and can cause production models to fail. These frameworks typically miss critical issues because they rely on aggregate metrics that don't capture the full picture. For example, fidelity tests often overlook the loss of correlation between features, while utility tests based on averages can hide poor performance on rare edge cases. The article proposes more robust evaluation methods, such as analyzing correlation matrices, stratifying utility tests by deciles, and performing attribute inference tests to better assess privacy risks. These improved techniques help ensure that synthetic data accurately represents the original data's complexity, especially in its tail distributions.