When EEG Models See the Same People in Training and Testing

Machine learning model performance on EEG data is significantly inflated when the same participants are present in both training and testing sets. A study comparing random-window splitting against a participant-held-out evaluation found substantial performance drops across logistic regression, SVM, and random forest models. This discrepancy arises because EEG spectral features contain strong participant-specific information, allowing models to identify individuals rather than generalize to the underlying task. The findings demonstrate that proper evaluation design, which holds out entire subjects, is critical for accurately assessing a model's ability to perform on new, unseen individuals.