MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
2023/04/20 by Deyao Zhu, Zhu, Deyao, Jun Chen +7 · 273 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2304.10592
Abstract
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
Cited by
- SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
- VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs
- Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark
- Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA
- Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
- LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos
- Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
- Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
- Group Preference Collapse in Personalized Multimodal Large Language Models
- MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence
- ORCA: Object Recognition and Comprehension for Archiving Marine Species
- MarineEval: Assessing the Marine Intelligence of Vision-Language Models
- Generative Digital Twins: Vision-Language Simulation Models for Executable Industrial Systems
- Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models
- ReasonCD: A Multimodal Reasoning Large Model for Implicit Change-of-Interest Semantic Mining
- Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
- Enhancing 3D Semantic Scene Completion with a Refinement Module
- HyDRA: Hierarchical and Dynamic Rank Adaptation for Mobile Vision Language Model
- A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
- 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
- Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
- PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
- AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
- Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in
- SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning
- MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
- VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
- Robust MLLM Unlearning via Visual Knowledge Distillation
- Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing
- DynaIP: Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generation
- Rethinking Chain-of-Thought Reasoning for Videos
- SATGround: A Spatially-Aware Approach for Visual Grounding in Remote Sensing
- InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs
- ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
- CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks
- OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation
- Qwen3.5-Omni Technical Report
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models
- Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
- Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilities
- GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes
- PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
- Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
- Understanding and Harnessing Sparsity in Unified Multimodal Models
- Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
- Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
- CauSight: Learning to Supersense for Visual Causal Discovery
- DiG-Flow: Discrepancy-Guided Flow Matching for Robust VLA Models
- DefenSee: Dissecting Threat from Sight and Text -- A Multi-View Defensive Pipeline for Multi-modal Jailbreaks
- M4-BLIP: Advancing Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis
- See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
- PhotoFramer: Multi-modal Image Composition Instruction
- Table as a Modality for Large Language Models
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning
- BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
- DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation
- Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
- SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
- SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
- Co-Training Vision Language Models for Remote Sensing Multi-task Learning
- Object-Centric Vision Token Pruning for Vision Language Models
- WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
- Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
- CLASH: A Benchmark for Cross-Modal Contradiction Detection
- Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
- Mixture of Horizons in Action Chunking
- ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Access
- Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
- SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
- Attention Guided Alignment in Efficient Vision-Language Models
- Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination
- Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
- Q-MLLM: Vector Quantization for Robust Multimodal Large Language Model Security
- VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
- Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration
- A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
- Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
- SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
- Jailbreaking Large Vision Language Models in Intelligent Transportation Systems
- Unlocking the Forgery Detection Potential of Vanilla MLLMs: A Novel Training-Free Pipeline
- Direct Visual Grounding by Directing Attention of Visual Tokens
- Suppressing VLM Hallucinations with Spectral Representation Filtering
- Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
- PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models
- EcoAlign: An Economically Rational Framework for Efficient LVLM Alignment
- Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
- Stroke Modeling Enables Vectorized Character Generation with Large Vectorized Glyph Model
- CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging
- Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation
- OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models
- AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
- VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models
- NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
- VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving
- Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
- Medical Referring Image Segmentation via Next-Token Mask Prediction
- DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding
- UniChange: Unifying Change Detection with Multimodal Large Language Model
- Text-guided Fine-Grained Video Anomaly Understanding
- NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding
- MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models
- CATCH: A Modular Cross-domain Adaptive Template with Hook
- Security Risk of Misalignment between Text and Image in Multi-modal Model
- Knowledge-Guided Textual Reasoning for Explainable Video Anomaly Detection via LLMs
- MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
- MemEIC: A Step Toward Continual and Compositional Knowledge Editing
- OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
- Large language model-based task planning for service robots: A review
- Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
- OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
- Unifying Large Language Models and Knowledge Graphs: A Roadmap
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
- Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images
- Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents
- MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
- Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning
- SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
- Spatial Preference Rewarding for MLLMs Spatial Understanding
- IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection
- Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytelling
- Salient Concept-Aware Generative Data Augmentation
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding
- InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
- OralGPT: A Two-Stage Vision-Language Model for Oral Mucosal Disease Diagnosis and Description
- Generative Universal Verifier as Multimodal Meta-Reasoner
- AgentCaster: Reasoning-Guided Tornado Forecasting
- CoDefend: Cross-Modal Collaborative Defense via Diffusion Purification and Prompt Optimization
- Learning Dynamics of VLM Finetuning
- RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluation
- Unified Open-World Segmentation with Multi-Modal Prompts
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
- CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
- Provable Training Data Identification for Large Language Models
- VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search
- Vision Language Models: A Survey of 26K Papers
- BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception
- VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commands
- CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning
- TTRV: Test-Time Reinforcement Learning for Vision Language Models
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
- MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
- UGround: Towards Unified Visual Grounding with Unrolled Transformers
- AgenticRAG: Tool-Augmented Foundation Models for Zero-Shot Explainable Recommender Systems
- Automated Structured Radiology Report Generation with Rich Clinical Context
- ProfVLM: A Lightweight Video-Language Model for Multi-View Proficiency Estimation
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
- Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Ranking
- LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
- Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
- Visual serial processing deficits explain divergences in human and VLM reasoning
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- SVAC: Scaling Is All You Need For Referring Video Object Segmentation
- ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generation
- Falcon: A Cross-Modal Evaluation Dataset for Comprehensive Safety Perception
- HIVTP: A Training-Free Method to Improve VLMs Efficiency via Hierarchical Visual Token Pruning Using Middle-Layer-Based Importance Score
- Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
- From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
- Benchmarking and Mitigating Sycophancy in Medical Vision Language Models
- Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization
- FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
- JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
- VCE: A Zero-Cost Hallucination Mitigation Method of LVLMs via Visual Contrastive Editing
- Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
- TransMI: a transfer-learning method for generalized map information evaluation
- Alternating Training-based Label Smoothing Enhances Prompt Generalization
- PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
- Knowledge Transfer from Interaction Learning
- Steering Multimodal Large Language Models Decoding for Context-Aware Safety
- The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographers
- DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
- Qwen3-Omni Technical Report
- MCTS-EP: Empowering Embodied Planning with Online Preference Optimization
- FitPro: A Zero-Shot Framework for Interactive Text-based Pedestrian Retrieval in Open World
- ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents
- Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generation
- Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
- ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models
- ORCA: Agentic Reasoning For Hallucination and Adversarial Robustness in Vision-Language Models
- Generalizable Geometric Image Caption Synthesis
- TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding
- A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
- MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
- Enhancing Video Large Language Models with Structured Multi-Video Collaborative Reasoning
- Open-ended Hierarchical Streaming Video Understanding with Vision Language Models
- Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
- Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding
- A Modular and Multimodal Generative AI Framework for Urban Building Energy Data: Generating Synthetic Homes
- A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answering
- Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
- Multimodal LLMs See Sentiment
- MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance
- Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model
- Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization
- Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
- Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model
- Sample-efficient Integration of New Modalities into Large Language Models
- VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality
- AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
- Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models
- Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
- Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
- Way to Build Native AI-driven 6G Air Interface: Principles, Roadmap, and Outlook
- GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
- MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
- How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
- DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking
- Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Models
- Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
- SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
- Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistance
- Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
- Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector
- Revisiting MLLM Token Technology through the Lens of Classical Visual Coding
- EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models
- Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
- MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
- KFFocus: Highlighting Keyframes for Enhanced Video Understanding
- Re:Verse -- Can Your VLM Read a Manga?
- TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimation
- AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning
- VL-MedGuide: A Visual-Linguistic Large Model for Intelligent and Explainable Skin Disease Auxiliary Diagnosis
- AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance
- SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images
- Training-Free Multimodal Large Language Model Orchestration
- Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
- Do Vision-Language Models Leak What They Learn? Adaptive Token-Weighted Model Inversion Attacks
- Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
- GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbations
- VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extraction
- VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
- SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation
- InspectVLM: Unified in Theory, Unreliable in Practice
- Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models
- SGCap: Decoding Semantic Group for Zero-shot Video Captioning
- Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
- GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models
- Adversarial-Guided Diffusion for Multimodal LLM Attacks
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
- Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
- ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
- AI in Agriculture: A Survey of Deep Learning Techniques for Crops, Fisheries and Livestock
- Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions
- METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model
- Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
- RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
- LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks
- Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
- A Survey of Token Compression for Efficient Multimodal Large Language Models
- Region-based Cluster Discrimination for Visual Representation Learning
- LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs
- Object-centric Video Question Answering with Visual Grounding and Referring
- BEV-LLM: Leveraging Multimodal BEV Maps for Scene Captioning in Autonomous Driving
- RemoteReasoner: Towards Unifying Geospatial Reasoning Workflow
- A Survey of Multimodal Hallucination Evaluation and Detection
- Q-Former Autoencoder: A Modern Framework for Medical Anomaly Detection
- Towards Effective Human-in-the-Loop Assistive AI Agents
- LMM-Det: Make Large Multimodal Models Excel in Object Detection
Related