2025/01/06 by Jun-Hong Shen, Junhong Shen, Kushal Tirumala +12 · 1 voice · 7 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Video Analysis and Summarization #cs.CV
paper · pdf · doi:10.48550/arxiv.2501.03120
openalex publication_date 2025/01/06 · arxiv published 2025/01/06 · arxiv updated 2025/01/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.