2024/12/11 by Ben Xue, Tan, Zhentao, Xue, Ben +14 · 4 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2412.10443
openalex publication_date 2024/12/11 · openalex created_date 2024/12/18 · openalex updated_date 2026/07/28
This paper presents the Semantic-aWarE spatial-tEmporal Tokenizer (SweetTok), a novel video tokenizer to overcome the limitations in current video tokenization methods for compacted yet effective discretization. Unlike previous approaches that process flattened local visual patches via direct discretization or adaptive query tokenization, SweetTok proposes a decoupling framework, compressing visual inputs through distinct spatial and temporal queries via Decoupled Query AutoEncoder (DQAE). This design allows SweetTok to efficiently compress video token count while achieving superior fidelity by capturing essential information across spatial and temporal dimensions. Furthermore, we design a Motion-enhanced Language Codebook (MLC) tailored for spatial and temporal compression to address the differences in semantic representation between appearance and motion information. SweetTok significantly improves video reconstruction results by 42.8% w.r.t rFVD on UCF-101 dataset. With a better token compression strategy, it also boosts downstream video generation results by 15.1% w.r.t gFVD. Additionally, the compressed decoupled tokens are imbued with semantic information, enabling few-shot recognition capabilities powered by LLMs in downstream applications.