BrainBind maps synchronized EEG-fMRI to long-form video embeddings on CineBrain with MRR 23.82 for fMRI under Omni
BrainBind demonstrates cross-modal retrieval between synchronized EEG and fMRI via long-form video embeddings on CineBrain. Metrics reveal modality-specific strengths: fMRI leads full-list organization while joint training improves top-rank precision. The work highlights privacy and inference implications for future brain-machine interfaces operating on naturalistic data.
The framework records EEG and fMRI simultaneously during extended naturalistic video stimuli rather than static images or short clips. It treats the resulting video embedding space as a shared semantic reference instead of attempting direct neural code reconstruction. Five-seed Recall@K matrices across two anchors and four configurations isolate retrieval profiles while an episode-disjoint split quantifies narrative context effects beyond stimulus identity.
CineBrain results show fMRI under Omni organizes full candidate lists most effectively while Joint-Dual achieves the highest label-aware R@1 at 13.52 percent with a flatter rank distribution. Proxy choice and system configuration produce distinct top-rank versus rank-wide behaviors. The paper reports code release but provides no external validation set or cross-dataset transfer metrics.
Integration of synchronized modalities at narrative scale directly informs brain-machine interface design by exposing how temporal and spatial resolutions interact under repeated semantic context. Privacy risks arise because embedding alignment can recover stimulus identity from either modality without explicit reconstruction. Mental health applications gain from better mapping of naturalistic narrative responses yet require safeguards against unintended inference of internal states.
Next steps include scaling to multi-subject synchronized corpora and testing whether embedding alignment generalizes to non-video stimuli such as auditory narratives or interactive tasks.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2611.05581)
- [2]Supporting Source(https://arxiv.org/abs/2306.07975)