Scaling Vision Transformers for Functional MRI with Flat Maps

2025/10/15 by Connor Lane, Lane, Connor, Tripathy, Mihir +19 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Biological sciences #FOS: Computer and information sciences #Image Processing Techniques and Applications #Image Processing and 3D Reconstruction #Medical Image Segmentation Techniques #Neurons and Cognition (q-bio.NC)

paper · pdf · doi:10.48550/arxiv.2510.13768

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

We study the problem of training self-supervised foundation models for functional MRI. Our main contributions are: (1) we introduce a new model family (CortexMAE) trained using the masked autoencoder framework on 2.1K hours of open fMRI data, and (2) we release the first open evaluation suite (Brainmarks) for fMRI foundation models. Our core innovation is simple: we adapt the Vision Transformer to fMRI by first converting each 3D fMRI volume to a 2D map using a cortical flat map projection. We directly compare flat maps to both parcellation and volume-based representations. While each has its advantages, flat maps generally perform best. We perform the first systematic scaling analysis for fMRI and observe strict power law scaling, albeit with limits. Finally, we use Brainmarks to do controlled benchmark comparisons. On subject-level trait prediction, we report a challenging null result: no single model achieves clear state-of-the-art performance. Moreover, all models struggle to outperform a simple functional connectivity baseline. On cognitive state decoding, we observe more robust performance, and in this setting our CortexMAE family outperforms prior models by a large margin. Code, models, and datasets are available at https://github.com/MedARC-AI/CortexMAE and https://github.com/MedARC-AI/Brainmarks.

Citations

• In Pursuit of Pixel Supervision for Visual Pre-training
• Brain-Semantoks: Learning Semantic Tokens of Brain Dynamics with a Self-Distilled Foundation Model
• Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens
• Towards a general-purpose foundation model for fMRI analysis
• Perception Encoder: The best visual embeddings are not at the output of the network
• MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
• Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking
• Spurious reconstruction from brain activity
• MindBridge: A Cross-Subject Brain Decoding Framework
• MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
• Rethinking Patch Dependence for Masked Autoencoders
• A Unified, Scalable Framework for Neural Population Decoding
• Vision Transformers Need Registers
• SwiFT: Swin 4D fMRI Transformer
• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
• Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
• Cortical analysis of heterogeneous clinical brain MRI scans for large-scale neuroimaging studies
• A Cookbook of Self-Supervised Learning
• DINOv2: Learning Robust Visual Features without Supervision
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
• Natural scene reconstruction from fMRI signals using generative latent diffusion
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --
• FlexiViT: One Model for All Patch Sizes
• Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding
• Brain Network Transformer
• Masked Autoencoders for Low dose CT denoising
• Decoding speech perception from non-invasive brain recordings
• Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data
• On Data Scaling in Masked Image Modeling
• Masked Autoencoders As Spatiotemporal Learners
• Training Compute-Optimal Large Language Models
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
• Self-Supervised Transformers for fMRI representation
• Masked Autoencoders Are Scalable Vision Learners
• On the Opportunities and Risks of Foundation Models
• Brain charts for the human lifespan
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• ViViT: A Video Vision Transformer
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Language Models are Few-Shot Learners
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Fine-grain atlases of functional modes for fMRI analysis
• Scaling Laws for Neural Language Models
• Topographic organization of the human subcortex unveiled with functional connectivity gradients
• SciPy 1.0: fundamental algorithms for scientific computing in Python
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Generic decoding of seen and imagined objects using hierarchical visual features
• Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity
• The Brain's Default Mode Network
• Microsoft COCO: Common Objects in Context
• Human Connectome Project informatics: Quality control, database services, and data visualization
• ImageNet classification with deep convolutional neural networks
• Distinctive Image Features from Scale-Invariant Keypoints
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Real-time Reconstruction of Human Visual Perception from fMRI

Related

• See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding
• Self-Supervised Transformers for fMRI representation
• Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding
• Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI