VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

2024/03/01 by Chu, Xiangxiang, Su, Jianlin, Zhang, Bo +1 · 16 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2403.00522

Abstract

Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding. Our code is released at https://github.com/Meituan-AutoML/VisionLLaMA.

Cited by

• Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models
• Next-Embedding Prediction Makes Strong Vision Learners
• GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
• Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General
• Eevee: Towards Close-up High-resolution Video-based Virtual Try-on
• VisPlay: Self-Evolving Vision-Language Models from Images
• A Circular Argument : Does RoPE need to be Equivariant for Vision?
• Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
• ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraints
• From Editor to Dense Geometry Estimator
• A Novel Framework for Automated Explain Vision Model Using Vision-Language Models
• Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models
• Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation
• PixNerd: Pixel Neural Field Diffusion
• GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures

Related

• Adapting LLaMA Decoder to Vision Transformer
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
• SOLO: A Single Transformer for Scalable Vision-Language Modeling
• Rethinking Visual Information Processing in Multimodal LLMs
• VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework