FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed

2025/07/04 by Jiaqi Zhang, Zhang, Jiaqi, Juntuo Wang +6
Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Experimental Learning in Engineering #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.03779

openalex publication_date 2025/07/04 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. But numerous scenarios require practitioners to reproduce those pre-training solutions, such as on private data, new modalities, or simply for scientific questioning--which is currently extremely demanding computation-wise. We thus propose a novel pre-training strategy for DINOv2 that simultaneously accelerates convergence--and strengthens robustness to common corruptions as a by-product. Our approach involves a frequency filtering curriculum--low-frequency being seen first--and the Gaussian noise patching augmentation. Applied to a ViT-B/16 backbone trained on ImageNet-1K, while pre-training time and FLOPs are reduced by 1.6x and 2.25x, our method still achieves matching robustness in corruption benchmarks (ImageNet-C) and maintains competitive linear probing performance compared with baseline. This dual benefit of efficiency and robustness makes large-scale self-supervised foundation modeling more attainable, while opening the door to novel exploration around data curriculum and augmentation as means to improve self-supervised learning models robustness. The code is available at https://github.com/KevinZ0217/fast_dinov2

Citations

• Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
• Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
• Robust Bird's Eye View Segmentation by Adapting DINOv2
• Mitigating Low-Frequency Bias: Feature Recalibration and Frequency Attention Regularization for Adversarial Robustness
• EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training
• Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks
• Towards Building More Robust Models with Frequency Bias
• A Cookbook of Self-Supervised Learning
• DINOv2: Learning Robust Visual Features without Supervision
• RECLIP: Resource-efficient CLIP by Training with Small Images
• LAION-5B: An open large-scale dataset for training next generation image-text models
• How Does Frequency Bias Affect the Robustness of Neural Image Classifiers against Common Corruption and Adversarial Perturbations?
• Understanding The Robustness in Vision Transformers
• How Do Vision Transformers Work?
• Masked Autoencoders Are Scalable Vision Learners
• Vision Transformers are Robust Learners
• Emerging Properties in Self-Supervised Vision Transformers
• Self-Supervised Pretraining Improves Self-Supervised Pretraining
• Learning Transferable Visual Models From Natural Language Supervision
• Adversarial Momentum-Contrastive Pre-Training
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Bootstrap your own latent: A new approach to self-supervised Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data Augmentation in Training CNNs: Injecting Noise to Images
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• A Fourier Perspective on Model Robustness in Computer Vision
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian\n Augmentation
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Improved Regularization of Convolutional Neural Networks with Cutout
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Matching Networks for One Shot Learning
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge

Related

• DINOv2: Learning Robust Visual Features without Supervision
• DINO-MX: A Modular & Flexible Framework for Self-Supervised Learning
• EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones
• Accessing Vision Foundation Models via ImageNet-1K
• Training state-of-the-art pathology foundation models with orders of magnitude less data