2024/03/03 by Kun-Yu Lin, Henghui Ding, Lin, Kun-Yu +12 · 1 citation
Computer Science · Health Professions · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Interpreting and Communication in Healthcare #Natural Language Processing Techniques #Text Readability and Simplification
paper · pdf · doi:10.48550/arxiv.2403.01560
openalex publication_date 2024/03/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Inspired by the impressive success of image-text foundation models, recent works have proposed to adapt these foundation models to video data, leading to efficient and effective video models for open-vocabulary action recognition. However, through a comprehensive evaluation, our work finds that state-of-the-art open-vocabulary action recognition models still struggle with generalization to video domains that they have not encountered. To address this limitation, we introduce generalizable open-vocabulary action recognition, which aims to develop action recognition models capable of generalizing to both novel action categories and unseen video domains. Our work contributes a novel model named XOV-Action to overcome two critical challenges: (1) understanding novel action concepts of open-set categories, and (2) mitigating the scenario discrepancy between training and test datasets. Specifically, XOV-Action first proposes to capture diverse action-related concepts by learning diversified elaboration representations, which enables better generalization to open-set action categories. Second, XOV-Action learns scene-agnostic video representations to overcome the scene bias, which improves the generalization in unseen video domains. Additionally, to evaluate models in generalizable open-vocabulary action recognition, we contribute a new cross-domain action benchmark named XOVABench, which covers multiple video domains with varying degrees of gaps and consists of both closed-set and open-set action categories. Extensive quantitative and qualitative experiments demonstrate that our proposed XOV-Action can effectively improve action recognition performance for both closed-set and open-set categories across video domains. The benchmark is available at https://github.com/KunyuLin/XOV-Action/.