2025/09/24 by Babak Naderi, Naderi, Babak, Ross Cutler +1
Computer Science · #FOS: Computer and information sciences #Image and Video Quality Assessment #Multimedia (cs.MM) #Video Coding and Compression Technologies #Visual Attention and Saliency Detection
paper · pdf · doi:10.48550/arxiv.2509.20118
openalex publication_date 2025/09/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In crowdsourced subjective video quality assessment, practitioners often face a choice between Absolute Category Rating (ACR), ACR with Hidden Reference (ACR-HR), and Comparison Category Rating (CCR). We conducted a P.910-compliant, side-by-side comparison across six studies using 15 talking-head sources of good and fair quality, processed with realistic degradations (blur, scaling, compression, freezing, and their combinations), as well as a practical bitrate-ladder task at 720p and 1080p resolutions. We evaluated statistical efficiency (standard deviations), economic efficiency, and decision agreement. Our results show that ACR-HR and ACR correlate strongly at the condition level, while CCR is more sensitive-capturing improvements beyond the reference. ACR-HR, however, exhibits compressed scale use, particularly for videos with fair source quality. ACR-HR is approximately twice as fast and cost-effective, with lower normalized variability, yet the choice of quality measurement method shifts saturation points and bitrate-ladder recommendations. Finally, we provide practical guidance on when to use each test method.