VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

2024/03/01 by Xiangxiang Chu, Jianlin Su, Chu, Xiangxiang +5 · 20 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #CCD and CMOS Imaging Sensors #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.2403.00522

openalex publication_date 2024/03/01 · openalex created_date 2024/03/05 · openalex updated_date 2026/07/28

Abstract

Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding. Our code is released at https://github.com/Meituan-AutoML/VisionLLaMA.

Cited by

• High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models
• Next-Embedding Prediction Makes Strong Vision Learners
• GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
• Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General
• Eevee: Towards Close-up High-resolution Video-based Virtual Try-on
• VisPlay: Self-Evolving Vision-Language Models from Images
• A Circular Argument : Does RoPE need to be Equivariant for Vision?
• Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
• ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraints
• From Editor to Dense Geometry Estimator
• A Novel Framework for Automated Explain Vision Model Using Vision-Language Models
• Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models
• Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation
• PixNerd: Pixel Neural Field Diffusion
• GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures
• FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing
• Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
• PiT: Progressive Diffusion Transformer
• LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformers
• Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding

Related

• Adapting LLaMA Decoder to Vision Transformer
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
• SOLO: A Single Transformer for Scalable Vision-Language Modeling
• Rethinking Visual Information Processing in Multimodal LLMs
• VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework