Sequential Modeling Enables Scalable Learning for Large Vision Models

2023/12/01 by Yutong Bai, Xinyang Geng, Bai, Yutong +14 · 1 voice · 28 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.2312.00785

openalex publication_date 2023/12/01 · arxiv published 2023/12/01 · arxiv updated 2023/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources such as semantic segmentations and depth reconstructions without needing any meta-knowledge beyond the pixels. Once this wide variety of visual data (comprising 420 billion tokens) is represented as sequences, the model can be trained to minimize a cross-entropy loss for next token prediction. By training across various scales of model architecture and data diversity, we provide empirical evidence that our models scale effectively. Many different vision tasks can be solved by designing suitable visual prompts at test time.

Cited by

• UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
• Reloc-VGGT: Visual Re-localization with Geometry Grounded Transformer
• VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation
• A Study of Finetuning Video Transformers for Multi-view Geometry Tasks
• SFTok: Bridging the Performance Gap in Discrete Tokenizers
• Spherical Leech Quantization for Visual Tokenization and Generation
• Can TabPFN Compete with GNNs for Node Classification via Graph Tabularization?
• Genomic Next-Token Predictors are In-Context Learners
• Stroke Modeling Enables Vectorized Character Generation with Large Vectorized Glyph Model
• Rethinking Visual Intelligence: Insights from Video Pretraining
• Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models
• Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction
• ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
• Point Prompting: Counterfactual Tracking with Video Diffusion Models
• Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?
• REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
• EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory
• Visual Self-Refinement for Autoregressive Models
• Can World Models Benefit VLMs for World Dynamics?
• Personalized Vision via Visual In-Context Learning
• PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learning
• UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models
• Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation
• Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation
• Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model
• Stable Diffusion Models are Secretly Good at Visual In-Context Learning
• Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systems

Discussions

• Sequential Modeling Enables Scalable Learning for Large Vision Models [hn, 3 points, 0 comments]

Related

• Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
• SOLO: A Single Transformer for Scalable Vision-Language Modeling
• UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models
• Scalable Pre-training of Large Autoregressive Image Models
• Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers