2018/07/02 by Phuc Nguyen, Deva Ramanan, Nguyen, Phuc +3 · 4 citations
Computer Science · Decision Sciences · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Fault Detection and Control Systems #Scientific Measurement and Uncertainty Evaluation #VLSI and Analog Circuit Testing
paper · pdf · doi:10.48550/arxiv.1807.00493
openalex publication_date 2018/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Much recent work on visual recognition aims to scale up learning to massive, noisily-annotated datasets. We address the problem of scaling- up the evaluation of such models to large-scale datasets with noisy labels. Current protocols for doing so require a human user to either vet (re-annotate) a small fraction of the test set and ignore the rest, or else correct errors in annotation as they are found through manual inspection of results. In this work, we re-formulate the problem as one of active testing, and examine strategies for efficiently querying a user so as to obtain an accu- rate performance estimate with minimal vetting. We demonstrate the effectiveness of our proposed active testing framework on estimating two performance metrics, Precision@K and mean Average Precision, for two popular computer vision tasks, multi-label classification and instance segmentation. We further show that our approach is able to save significant human annotation effort and is more robust than alternative evaluation protocols.