Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory

2017/11/03 by Ron Amit, Amit, Ron, Ron Meir +1 · 6 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1711.01244

Accepted to ICML 2018

openalex publication_date 2017/11/03 · arxiv created 2019/05/20 · arxiv updated 2019/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

In meta-learning an agent extracts knowledge from observed tasks, aiming to facilitate learning of novel future tasks. Under the assumption that future tasks are 'related' to previous tasks, the accumulated knowledge should be learned in a way which captures the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of new tasks. We present a framework for meta-learning that is based on generalization error bounds, allowing us to extend various PAC-Bayes bounds to meta-learning. Learning takes place through the construction of a distribution over hypotheses based on the observed tasks, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting an experience-dependent prior for novel tasks. We develop a gradient-based algorithm which minimizes an objective function derived from the bounds and demonstrate its effectiveness numerically with deep neural networks. In addition to establishing the improved performance available through meta-learning, we demonstrate the intuitive way by which prior information is manifested at different levels of the network.

Citations

• PAC-Bayesian aggregation and multi-armed bandits
• A Note on the PAC Bayesian Theorem
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Auto-Encoding Variational Bayes
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• A PAC-Bayesian bound for Lifelong Learning
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Distral: Robust Multitask Reinforcement Learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Learning Multiple Layers of Features from Tiny Images

Cited by

• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL
• On Quantification of Borrowing of Information in Hierarchical Bayesian Models
• Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizes
• Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions
• A Cryptographic Perspective on Mitigation vs. Detection in Machine Learning
• Meta-Strategy for Learning Tuning Parameters with Guarantees

Related

• PAC-Bayes Bounds for Meta-learning with Data-Dependent Prior
• More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Meta-Learning in Neural Networks: A Survey
• A General framework for PAC-Bayes Bounds for Meta-Learning