Stochastic Siamese MAE Pretraining for Longitudinal Medical Images

2025/12/29 by Taha Emre, Emre, Taha, Arunava Chakravarty +21
Computer Science · Medicine · #Autoencoder #Computer Vision and Pattern Recognition (cs.CV) #Deep learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Feature learning #Generative Adversarial Networks and Image Synthesis #Inference #Machine Learning (cs.LG) #Pattern recognition (psychology) #Representation (politics) #Retinal Imaging and Analysis #Temporal resolution #Timestamp

paper · open access · doi:10.48550/arxiv.2512.23441

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/29 · openalex created_date 2025/12/31 · openalex updated_date 2026/07/28

Abstract

Temporally aware image representations are crucial for capturing disease progression in 3D volumes of longitudinal medical datasets. However, recent state-of-the-art self-supervised learning approaches like Masked Autoencoding (MAE), despite their strong representation learning capabilities, lack temporal awareness. In this paper, we propose STAMP (Stochastic Temporal Autoencoder with Masked Pretraining), a Siamese MAE framework that encodes temporal information through a stochastic process by conditioning on the time difference between the 2 input volumes. Unlike deterministic Siamese approaches, which compare scans from different time points but fail to account for the inherent uncertainty in disease evolution, STAMP learns temporal dynamics stochastically by reframing the MAE reconstruction loss as a conditional variational inference objective. We evaluated STAMP on two OCT and one MRI datasets with multiple visits per patient. STAMP pretrained ViT models outperformed both existing temporal MAE methods and foundation models on different late stage Age-Related Macular Degeneration and Alzheimer's Disease progression prediction which require models to learn the underlying non-deterministic temporal dynamics of the diseases.

Citations

• Accurate and Efficient World Modeling with Masked Latent Transformers
• Efficient World Models with Context-Aware Tokenization
• Visual Representation Learning with Stochastic Frame Prediction
• Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images
• Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge
• L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction
• Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing
• Scalable Pre-training of Large Autoregressive Image Models
• DUSt3R: Geometric 3D Vision Made Easy
• Learning to Act without Actions
• Understanding Masked Autoencoders via Hierarchical Latent Variable Models
• Mastering Diverse Domains through World Models
• Scalable Diffusion Models with Transformers
• CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
• Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis
• Masked Autoencoders As Spatiotemporal Learners
• Masked Image Modeling Advances 3D Medical Image Analysis
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• Context Autoencoder for Self-Supervised Representation Learning
• Masked Autoencoders Are Scalable Vision Learners
• Is Space-Time Attention All You Need for Video Understanding?
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Modeling Disease Progression In Retinal OCTs With Longitudinal Self-Supervised Learning
• Stochastic Video Generation with a Learned Prior
• Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
• Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
• Semi-Supervised Learning with Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• N4ITK: Improved N3 Bias Correction

Related

• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling
• L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction
• Modeling Disease Progression In Retinal OCTs With Longitudinal Self-Supervised Learning
• Spatiotemporal Representation Learning for Short and Long Medical Image Time Series
• MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting