Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

2022/11/13 by Zijiao Chen, Chen, Zijiao, Jiaxin Qing +7 · 16 citations
Neuroscience · Biochemistry, Genetics and Molecular Biology · #Functional Brain Connectivity Studies #Neural dynamics and brain function #Cell Image Analysis Techniques

paper · pdf · doi:10.48550/arxiv.2211.06956

Abstract

Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.

Cited by

• Symbiotic Brain-Machine Drawing via Visual Brain-Computer Interfaces
• Unveiling Deep Semantic Uncertainty Perception for Language-Anchored Multi-modal Vision-Brain Alignment
• ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding
• Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer
• Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction
• In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
• Scaling Vision Transformers for Functional MRI with Flat Maps
• Towards Interpretable Visual Decoding with Attention to Brain Representations
• iMIND: Insightful Multi-subject Invariant Neural Decoding
• Overcoming Output Dimension Collapse: When Sparsity Enables Zero-shot Brain-to-Image Reconstruction at Small Data Scales
• UMind: A Unified Multitask Network for Zero-Shot M/EEG Visual Decoding
• Rest2Visual: Predicting Visually Evoked fMRI from Resting-State Scans
• BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding
• SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning
• Non-asymptotic convergence bound of conditional diffusion models

Related

• Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion
• Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
• Towards Interpretable Visual Decoding with Attention to Brain Representations
• Natural scene reconstruction from fMRI signals using generative latent diffusion
• Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs