LlamaSeg: Image Segmentation via Autoregressive Mask Generation
2025/05/26 by Deng, Jiru, Weng, Tengjin, Yang, Tianyu +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2505.19422
Abstract
We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs. By adhering to the next-token prediction paradigm, our approach naturally integrates segmentation tasks into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open-vocabulary labels or diverse textual descriptions, covering a wide spectrum of real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. To more accurately evaluate the quality of masks produced by visual generative models, we further propose a composite metric that combines Intersection over Union (IoU) with Average Hausdorff Distance (AHD), offering a more precise assessment of contour fidelity. Experimental results demonstrate that our method surpasses existing generative models across multiple datasets and yields more detailed segmentation masks.
Citations
- Seg-VAR: Image Segmentation with Visual Autoregressive Modeling
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
- VisNumBench: Evaluating Number Sense of Multimodal Large Language Models
- HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Text4Seg: Reimagining Image Segmentation as Text Generation
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- LLaVA-OneVision: Easy Visual Task Transfer
- OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
- Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- LaSagnA: Language-based Segmentation Assistant for Complex Queries
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- GiT: Towards Generalist Vision Transformer through Universal Language Interface
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
- GSVA: Generalized Segmentation via Multimodal Large Language Models
- PixelLM: Pixel Reasoning with Large Multimodal Model
- u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
- GLaMM: Pixel Grounding Large Multimodal Model
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- LISA: Reasoning Segmentation via Large Language Model
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- GRES: Generalized Referring Expression Segmentation
- VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Visual Instruction Tuning
- Micrograph segmentations for DDEVD
- Segment Anything
- Sigmoid Loss for Language Image Pre-Training
- Generative Semantic Segmentation
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- LLaMA: Open and Efficient Foundation Language Models
- PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- All in Tokens: Unifying Output Space of Visual Tasks via Soft Token
- MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
- Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
- Flamingo: a Visual Language Model for Few-Shot Learning
- PaLM: Scaling Language Modeling with Pathways
- ReSTR: Convolution-free Referring Image Segmentation Using Transformers
- LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
- CRIS: CLIP-Driven Referring Image Segmentation
- Vector-quantized Image Modeling with Improved VQGAN
- Vision-Language Transformer and Query Generation for Referring Segmentation
- Per-Pixel Classification is Not All You Need for Semantic Segmentation
- LoRA: Low-Rank Adaptation of Large Language Models
- SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Learning Transferable Visual Models From Natural Language Supervision
- Zero-Shot Text-to-Image Generation
- Taming Transformers for High-Resolution Image Synthesis
- Language Models are Few-Shot Learners
- GLU Variants Improve Transformer
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Root Mean Square Layer Normalization
- Generating Diverse High-Fidelity Images with VQ-VAE-2
- Neural Discrete Representation Learning
- Attention Is All You Need
- COCO-Stuff: Thing and Stuff Classes in Context
- Semantic Understanding of Scenes through the ADE20K Dataset
- Semantic Understanding of Scenes Through the ADE20K Dataset
- Modeling Context in Referring Expressions
- Gaussian Error Linear Units (GELUs)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Related