2017/11/02 by Lei Han, Han, Lei, Yiheng Huang +3
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1711.00658
Published in ICML 2018
openalex publication_date 2017/11/02 · arxiv created 2018/09/13 · arxiv updated 2018/09/14 · openalex created_date 2019/07/30 · openalex updated_date 2026/07/28
This paper proposes a method for multi-class classification problems, where the number of classes K is large. The method, referred to as Candidates vs. Noises Estimation (CANE), selects a small subset of candidate classes and samples the remaining classes. We show that CANE is always consistent and computationally efficient. Moreover, the resulting estimator has low statistical variance approaching that of the maximum likelihood estimator, when the observed label belongs to the selected candidates with high probability. In practice, we use a tree structure with leaves as classes to promote fast beam search for candidate selection. We further apply the CANE method to estimate word probabilities in learning large neural language models. Extensive experimental results show that CANE achieves better prediction accuracy over the Noise-Contrastive Estimation (NCE), its variants and a number of the state-of-the-art tree classifiers, while it gains significant speedup compared to standard O(K) methods.