Self-supervised self-supervision by combining deep learning and probabilistic logic

2020/12/23 by Hunter Lang, Hoifung Poon, Lang, Hunter +1
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.12474

openalex publication_date 2020/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Labeling training examples at scale is a perennial challenge in machine learning. Self-supervision methods compensate for the lack of direct supervision by leveraging prior knowledge to automatically generate noisy labeled examples. Deep probabilistic logic (DPL) is a unifying framework for self-supervised learning that represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. While DPL is successful at combining pre-specified self-supervision, manually crafting self-supervision to attain high accuracy may still be tedious and challenging. In this paper, we propose Self-Supervised Self-Supervision (S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial "seed," S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments show that S4 is able to automatically propose accurate self-supervision and can often nearly match the accuracy of supervised methods with a tiny fraction of the human effort.

Citations

• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Semi-Supervised Learning with Deep Generative Models
• Composing graphical models with neural networks for structured representations and fast inference
• Incidental Supervision: Moving beyond Supervised Learning
• Adaptive Rule Discovery for Labeling Text Data

Related

• Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning
• Deep Probabilistic Logic: A Unifying Framework for Indirect Supervision
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• Self-supervision of Feature Transformation for Further Improving Supervised Learning
• Better Self-training for Image Classification through Self-supervision