PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

2025/04/17 by Jang Hyun Cho, Andrea Madotto, Cho, Jang Hyun +54 · 30 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2504.13180

openalex publication_date 2025/04/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost of measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding. We analyze standard training pipelines without distillation from proprietary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. To bridge these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. Additionally, we introduce PLM-VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. We make our work fully reproducible by providing data, training recipes, code & models. https://github.com/facebookresearch/perception_models

Cited by

• Same or Not? Enhancing Visual Perception in Vision-Language Models
• Revisiting the Platonic Representation Hypothesis: An Aristotelian View
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
• Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in
• VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
• What matters for Representation Alignment: Global Information or Spatial Structure?
• Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
• Video-CoM: Interactive Video Reasoning via Chain of Manipulations
• Text-Guided Semantic Image Encoder
• NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering
• EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• DigiData: Training and Evaluating General-Purpose Mobile Control Agents
• Visual Spatial Tuning
• What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenes
• Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence
• Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
• FineVision: Open Data Is All You Need
• InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
• HoneyBee: Data Recipes for Vision-Language Reasoners
• VUGEN: Visual Understanding priors for GENeration
• When Thinking Drifts: Evidential Grounding for Robust Video Reasoning
• From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• Multi Anatomy X-Ray Foundation Model
• Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data
• Planning with Reasoning using Vision Language World Model
• Group Relative Augmentation for Data Efficient Action Detection

Related

• GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
• Perception Encoder: The best visual embeddings are not at the output of the network
• VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
• Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs