Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion

2024/11/14 by Matteo Ferrante, Ferrante, Matteo, Tommaso Boccato +5 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2411.09723

openalex publication_date 2024/11/14 · openalex created_date 2024/11/21 · openalex updated_date 2026/07/28

Abstract

This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, illustrating its potential in decoding, encoding, and modality conversion tasks.

Cited by

• A Dough-Like Model for Understanding Double-Slit Phenomena
• Towards Interpretable Visual Decoding with Attention to Brain Representations
• Foundation Models for Cross-Domain EEG Analysis Application: A Survey
• On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
• Bridging Brain with Foundation Models through Self-Supervised Learning
• Transformer brain encoders explain human high-level visual responses

Related

• Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding
• BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings
• Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features
• Interpretable EEG-to-Image Generation with Semantic Prompts
• Modality-Agnostic fMRI Decoding of Vision and Language