Probing the State of the Art: A Critical Look at Visual Representation Evaluation

2019/11/30 by Cinjon Resnick, Resnick, Cinjon, Zeping Zhan +3
Computer Science · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1912.00215

erd59xH@Rqt!XCMsCmnz

arxiv created 2021/08/12 · arxiv updated 2021/08/13

Abstract

Self-supervised research improved greatly over the past half decade, with much of the growth being driven by objectives that are hard to quantitatively compare. These techniques include colorization, cyclical consistency, and noise-contrastive estimation from image patches. Consequently, the field has settled on a handful of measurements that depend on linear probes to adjudicate which approaches are the best. Our first contribution is to show that this test is insufficient and that models which perform poorly (strongly) on linear classification can perform strongly (weakly) on more involved tasks like temporal activity localization. Our second contribution is to analyze the capabilities of five different representations. And our third contribution is a much needed new dataset for temporal activity localization.

Related

• Visual Representations: Defining Properties and Deep Approximations
• Scaling and Benchmarking Self-Supervised Visual Representation Learning
• Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data
• Rethinking Evaluation Protocols of Visual Representations Learned via Self-supervised Learning
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark