Robust Learning from Untrusted Sources

2019/01/29 by Nikola Konstantinov, Christoph H. Lampert, Konstantinov, Nikola +2 · 7 citations
Computer Science · Mathematics · #Distributed Sensor Networks and Detection Algorithms #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1901.10310

Accepted to International Conference on Machine Learning (ICML), 2019; Camera-ready version

arxiv created 2019/05/17 · arxiv updated 2019/05/20

Abstract

Modern machine learning methods often require more data for training than a single expert can provide. Therefore, it has become a standard procedure to collect data from external sources, e.g. via crowdsourcing. Unfortunately, the quality of these sources is not always guaranteed. As additional complications, the data might be stored in a distributed way, or might even have to remain private. In this work, we address the question of how to learn robustly in such scenarios. Studying the problem through the lens of statistical learning theory, we derive a procedure that allows for learning from all available sources, yet automatically suppresses irrelevant or corrupted data. We show by extensive experiments that our method provides significant improvements over alternative approaches from robust statistics and distributed optimization.

Cited by

• Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning
• On the Sample Complexity of Adversarial Multi-Source PAC Learning
• Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Models
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• A Theory of Multiple-Source Adaptation with Limited Target Labeled Data
• Federated Adapter on Foundation Models: An Out-Of-Distribution Approach
• A Discriminative Technique for Multiple-Source Adaptation

Related

• Learning from Untrusted Data
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data
• Distributionally Robust Learning for Multi-source Unsupervised Domain Adaptation
• RobustFed: A Truth Inference Approach for Robust Federated Learning
• Probabilistic Inference for Learning from Untrusted Sources