Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding

2026/05/31 by Xiaoyang Li, Zeyan Tao
Engineering · Computer Science · Biochemistry, Genetics and Molecular Biology · #eess.SP #cs.CL #cs.CV #cs.LG #cs.SD #q-bio.NC

paper · pdf

19 pages, 7 figures; includes 11-page supplementary material. Associated code, prediction records, source data, and reproducibility materials: https://doi.org/10.5281/zenodo.21805983

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

We tested whether auditory-evoked EEG supports subject-independent five-vowel perception decoding when trial identity, model identity, prediction provenance, and participant-level inference are controlled within a single benchmark. We reconstructed Study 2 event tables from OpenNeuro ds006104 version 1.0.1 and analyzed the consonant-vowel pair task. One-to-one marker-stimulus pairing yielded 3,840 independent trials; control-condition selection and artifact rejection retained 1,094 epochs from 16 participants and 61 EEG channels. Thirteen unique implementations were evaluated using leave-one-subject-out testing, with participant metrics reconstructed from 36,102 trial predictions across 33 complete prediction replicas. Random Forest was numerically highest at 21.474% balanced accuracy (95% participant-bootstrap interval, 19.526-23.482%; chance, 20%), but neither its participant-level tests nor any implementation survived correction across the 13-model family. Deep-model performance was close to chance, and several architectures showed substantial seed-dependent variation and low trial-label agreement. In a separate descriptive sensor-space representation, participant-associated effects accounted for 72.24% of the balanced standardized centroid sum of squares, compared with 2.04% for vowel-associated effects; between-participant same-vowel distances exceeded within-participant across-vowel distances for all 16 participants. An exploratory MDM analysis comprising 9,616 genuine refits across training cohorts of 3-15 participants showed no monotonic performance gain. Within this dataset and protocol, evidence for reliable cross-subject five-vowel decoding is limited. The benchmark provides a reproducible chain from source rows to retained epochs, predictions, participant-level metrics, multiplicity-adjusted inference, and bounded diagnostic analyses.

Citations

• How EEG preprocessing shapes decoding performance
• EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization
• Decoding speech perception from non-invasive brain recordings
• Deep learning-based electroencephalography analysis: a systematic review
• Transfer Learning for Brain-Computer Interfaces: A Euclidean Space Data Alignment Approach
• MOABB: trustworthy algorithm benchmarking for BCIs
• Cross-validation failure: small sample sizes lead to large error bars
• Deep learning with convolutional neural networks for EEG decoding and visualization
• EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces
• Assessing and tuning brain decoders: cross-validation, caveats, and guidelines
• A well-conditioned estimator for large-dimensional covariance matrices