2017/11/01 by Mostafa Dehghani, Aliaksei Severyn, Dehghani, Mostafa +5 · 5 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.1711.00313
openalex publication_date 2017/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Training deep neural networks requires massive amounts of training data, but\nfor many tasks only limited labeled data is available. This makes weak\nsupervision attractive, using weak or noisy signals like the output of\nheuristic methods or user click-through data for training. In a semi-supervised\nsetting, we can use a large set of data with weak labels to pretrain a neural\nnetwork and then fine-tune the parameters with a small amount of data with true\nlabels. This feels intuitively sub-optimal as these two independent stages\nleave the model unaware about the varying label quality. What if we could\nsomehow inform the model about the label quality? In this paper, we propose a\nsemi-supervised learning method where we train two neural networks in a\nmulti-task fashion: a "target network" and a "confidence network". The target\nnetwork is optimized to perform a given task and is trained using a large set\nof unlabeled data that are weakly annotated. We propose to weight the gradient\nupdates to the target network using the scores provided by the second\nconfidence network, which is trained on a small amount of supervised data. Thus\nwe avoid that the weight updates computed from noisy labels harm the quality of\nthe target network model. We evaluate our learning strategy on two different\ntasks: document ranking and sentiment classification. The results demonstrate\nthat our approach not only enhances the performance compared to the baselines\nbut also speeds up the learning process from weak labels.\n