A Cookbook of Self-Supervised Learning

2023/04/24 by Randall Balestriero, Mark Ibrahim, Balestriero, Randall +36 · 2 voices · 32 citations
Computer Science · #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2304.12210

Abstract

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Cited by

• Understanding neural circuit principles for representation learning through joint-embedding predictive architectures
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime
• Evolutionary Neural Architecture Search with Dual Contrastive Learning
• Supervised Contrastive Frame Aggregation for Video Representation Learning
• Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning
• stable-pretraining-v1: Foundation Model Research Made Simple
• Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift
• Investigating self-supervised representations for audio-visual deepfake detection
• Image Hashing via Cross-View Code Alignment in the Age of Foundation Models
• FlexICL: A Flexible Visual In-context Learning Framework for Elbow and Wrist Ultrasound Segmentation
• Eigenfunction Extraction for Ordered Representation Learning
• SARATR-X: Toward Building a Foundation Model for SAR Target Recognition
• Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology
• Scaling Vision Transformers for Functional MRI with Flat Maps
• FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data
• Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detection
• Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density
• ActiveMark: on watermarking of visual foundation models via massive activations
• Self-Supervised Representation Learning as Mutual Information Maximization
• Directed Information γ-covering: An Information-Theoretic Framework for Context Engineering
• Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
• An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection
• JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
• On the Status of Foundation Models for SAR Imagery
• Three Pillars improving Vision Foundation Model Distillation for Lidar
• Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairs
• Enhancing Self-Supervised Speaker Verification Using Similarity-Connected Graphs and GCN
• Self-supervised neural operator for solving partial differential equations
• Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation
• CITADEL: Continual Anomaly Detection for Enhanced Learning in IoT Intrusion Detection
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• Self-supervised learning [wikipedia]

Discussions

• A Cookbook of Self-Supervised Learning [hn, 178 points, 22 comments]
• A Cookbook of Self-Supervised Learning [hn, 5 points, 1 comments]

Related

• Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
• Reverse Engineering Self-Supervised Learning
• A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends
• FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learning
• SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning