OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

2022/02/07 by Peng Wang, Yang An, Wang, Peng +17 · 34 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2202.03052

openalex publication_date 2022/02/07 · openalex created_date 2022/06/13 · openalex updated_date 2026/07/28

Abstract

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.

Cited by

• LLM-Free Image Captioning Evaluation in Reference-Flexible Settings
• Grounding Everything in Tokens for Multimodal Large Language Models
• CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation
• MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding
• Artemis: Structured Visual Reasoning for Perception Policy Learning
• OmniFD: A Unified Model for Versatile Face Forgery Detection
• MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
• ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training
• FOCUS: Efficient Keyframe Selection for Long Video Understanding
• Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction
• MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning
• Vision-Centric Activation and Coordination for Multimodal Large Language Models
• Vision Language Models: A Survey of 26K Papers
• Referring Expression Comprehension for Small Objects
• VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
• Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Representations for Language-based Object Detection
• UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
• AToken: A Unified Tokenizer for Vision
• A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
• Improving Generalized Visual Grounding with Instance-aware Joint Learning
• Multi Anatomy X-Ray Foundation Model
• CLAIRE: A Dual Encoder Network with RIFT Loss and Phi-3 Small Language Model Based Interpretability for Cross-Modality Synthetic Aperture Radar and Optical Land Cover Segmentation
• Towards Understanding Visual Grounding in Visual Language Models
• Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding
• DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking
• ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
• Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
• A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection
• A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understanding
• ImpliHateVid: A Benchmark Dataset and Two-stage Contrastive Learning Framework for Implicit Hate Speech Detection in Videos
• FrEVL: Leveraging Frozen Pretrained Embeddings for Efficient Vision-Language Understanding
• Phi-Ground Tech Report: Advancing Perception in GUI Grounding
• Beyond Class Tokens: LLM-guided Dominant Property Mining for Few-shot Classification
• LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

Related

• OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models
• u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
• Explore the Limits of Omni-modal Pretraining at Scale
• Everything is a Video: Unifying Modalities through Next-Frame Prediction
• UnIVAL: Unified Model for Image, Video, Audio and Language Tasks