2020/02/10 by Ross Cutler, Ramin Mehran, Cutler, Ross +11 · 1 citation
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Indoor and Outdoor Localization Technologies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimedia (cs.MM) #Music and Audio Processing #Speech and Audio Processing #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2002.03977
openalex publication_date 2020/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Active speaker detection (ASD) and virtual cinematography (VC) can\nsignificantly improve the remote user experience of a video conference by\nautomatically panning, tilting and zooming of a video conferencing camera:\nusers subjectively rate an expert video cinematographer's video significantly\nhigher than unedited video. We describe a new automated ASD and VC that\nperforms within 0.3 MOS of an expert cinematographer based on subjective\nratings with a 1-5 scale. This system uses a 4K wide-FOV camera, a depth\ncamera, and a microphone array; it extracts features from each modality and\ntrains an ASD using an AdaBoost machine learning system that is very efficient\nand runs in real-time. A VC is similarly trained using machine learning to\noptimize the subjective quality of the overall experience. To avoid distracting\nthe room participants and reduce switching latency the system has no moving\nparts -- the VC works by cropping and zooming the 4K wide-FOV video stream. The\nsystem was tuned and evaluated using extensive crowdsourcing techniques and\nevaluated on a dataset with N=100 meetings, each 2-5 minutes in length.\n