Rethinking supervised learning: insights from biological learning and from calling it by its name

2020/12/04 by Alex Hernandez-Garcia, Álex Hernández-García, Hernandez-Garcia, Alex
Biochemistry, Genetics and Molecular Biology · Computer Science · Psychology · #Artificial intelligence #Artificial neural network #Biomedical Text Mining and Ontologies #Cognitive science #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Data science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine learning #Multimodal Machine Learning Applications #Psychology #Supervised learning #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.02526

published in arXiv (Cornell University) (Cornell University) · Perspective paper. 8 pages + references. Earlier, shorter version accepted at the workshop SVRHM, NeurIPS 2020

openalex publication_date 2020/12/04 · arxiv created 2021/06/22 · arxiv updated 2021/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The renaissance of artificial neural networks was catalysed by the success of classification models, tagged by the community with the broader term supervised learning. The extraordinary results gave rise to a hype loaded with ambitious promises and overstatements. Soon the community realised that the success owed much to the availability of thousands of labelled examples and supervised learning went, for many, from glory to shame: Some criticised deep learning as a whole and others proclaimed that the way forward had to be alternatives to supervised learning: predictive, unsupervised, semi-supervised and, more recently, self-supervised learning. However, all these seem brand names, rather than actual categories of a theoretically grounded taxonomy. Moreover, the call to banish supervised learning was motivated by the questionable claim that humans learn with little or no supervision and are capable of robust out-of-distribution generalisation. Here, we review insights about learning and supervision in nature, revisit the notion that learning and generalisation are not possible without supervision or inductive biases and argue that we will make better progress if we just call it by its name.

Citations

• Unsupervised Visual Representation Learning by Context Prediction
• Looking at upside-down faces.
• The perceptron: A probabilistic model for information storage and organization in the brain.
• The Lack of A Priori Distinctions Between Learning Algorithms
• ImageNet Large Scale Visual Recognition Challenge
• A Closer Look at Memorization in Deep Networks
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Deep Learning: A Critical Appraisal
• Data augmentation instead of explicit regularization
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Shortcut learning in deep neural networks
• A Comprehensive Survey on Transfer Learning

Related

• Is 'Unsupervised Learning' a Misconceived Term?
• On the surprising similarities between supervised and self-supervised models
• Learning and Unlearning: Bridging classification, memory and generative modeling in Recurrent Neural Networks
• Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
• Deep Learning in Neural Networks: An Overview