No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

2024/07/15 by Walter Simoncini, Simoncini, Walter, Spyros Gidaris +6 · 1 voice · 2 citations
Computer Science · Earth and Planetary Sciences · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Enhancement Techniques #Machine Learning (cs.LG) #Underwater Acoustics Research #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2407.10964

openalex publication_date 2024/07/15 · arxiv published 2024/07/15 · arxiv updated 2024/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.

Cited by

• DIP: Unsupervised Dense In-Context Post-training of Visual Representations
• V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
• Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offs

Discussions

• Links to all papers: FUNGI: arxiv.org/abs/2407.10964 ManiPose: arxiv.org/abs/2312.06386 aMCL: arxiv.org/abs/2407.15580 GEPS: arxiv.org/abs/2410.23889 Concept XAI: arxiv.org/abs/2406.08074 DiffCut: arx [bsky, 2 points, 0 comments]

Related

• Analyzing Local Representations of Self-supervised Vision Transformers
• Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning
• MST: Masked Self-Supervised Transformer for Visual Representation
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models