Improved Baselines with Visual Instruction Tuning

2023/10/05 by Liu, Haotian, Li, Chunyuan, Li, Yuheng +1 · 709 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2310.03744

Abstract

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.

Cited by

• Instruction-Following Evaluation of Large Vision-Language Models
• CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generation
• GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation
• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
• Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference
• GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark
• Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA
• Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
• MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning
• The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models
• Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigation
• DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
• UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
• The Spatial Blindspot of Vision-Language Models
• Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
• Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
• Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
• Teaching Tiny VLA Models Where to Look and How to Move
• A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions
• Detect Before You Leap: Mirage Detection in Vision-Language Models
• Group Preference Collapse in Personalized Multimodal Large Language Models
• Compressing LLMs with MoP: Mixture of Pruners
• Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?
• Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models
• ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
• RLLaVA: An RL-central Framework for Language and Vision Assistants
• Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models
• Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks
• Input-Adaptive Visual Preprocessing for Efficient Fast Vision-Language Model Inference
• FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
• How I Met Your Bias: Investigating Bias Amplification in Diffusion Models
• SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images
• Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
• Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
• Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
• Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing
• Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• Enhancing 3D Semantic Scene Completion with a Refinement Module
• FPBench: A Comprehensive Benchmark of Multimodal Large Language Models for Fingerprint Analysis
• Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images
• 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• AdaTooler-V: Adaptive Tool-Use for Images and Videos
• PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
• Do Multi-Agents Solve Better Than Single? Evaluating Agentic Frameworks for Diagram-Grounded Geometry Problem Solving and Reasoning
• SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
• Collaborative Edge-to-Server Inference for Vision-Language Models
• AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
• Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain
• Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
• JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction
• Unified Semantic Transformer for 3D Scene Understanding
• Optimizing the Adversarial Perturbation with a Momentum-based Adaptive Matrix
• Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes
• SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning
• AgentIAD: Agentic Industrial Anomaly Detection via Adaptive Memory Augmentation
• Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
• Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making
• From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation
• Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
• DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
• EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
• VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
• Image Tiling for High-Resolution Reasoning: Balancing Local Detail with Global Context
• VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction
• Interpretable and Steerable Concept Bottleneck Sparse Autoencoders
• AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence
• Beyond Pixels: A Training-Free, Text-to-Text Framework for Remote Sensing Image Retrieval
• Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• VisualActBench: Can VLMs See and Act like a Human?
• Rethinking Chain-of-Thought Reasoning for Videos
• DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generation
• Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
• SATGround: A Spatially-Aware Approach for Visual Grounding in Remote Sensing
• Beyond Real Weights: Hypercomplex Representations for Stable Quantization
• HybridToken-VLM: Hybrid Token Compression for Vision-Language Models
• Relational Visual Similarity
• SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs
• START: Spatial and Textual Learning for Chart Understanding
• ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
• CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation
• Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
• ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior
• ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education
• Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models
• MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• ResearchArcade: Graph Interface for Academic Tasks
• EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture
• Towards Cross-View Point Correspondence in Vision-Language Models
• Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shot
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
• Jina-VLM: Small Multilingual Vision Language Model
• C3G: Learning Compact 3D Representations with 2K Gaussians
• Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning
• Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation
• ViDiC: Video Difference Captioning
• Fairness-Aware Fine-Tuning of Vision-Language Models for Medical Glaucoma Diagnosis
• Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval
• OneThinker: All-in-one Reasoning Model for Image and Video
• Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilities
• MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
• FiMMIA: scaling semantic perturbation-based membership inference across modalities
• PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
• GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
• SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts
• Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction
• GeoDiT: A Diffusion-based Vision-Language Model for Geospatial Understanding
• See, Think, Learn: A Self-Taught Multimodal Reasoner
• VACoT: Rethinking Visual Data Augmentation with VLMs
• VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
• Artemis: Structured Visual Reasoning for Perception Policy Learning
• Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
• DiG-Flow: Discrepancy-Guided Flow Matching for Robust VLA Models
• LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
• InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
• See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
• FOM-Nav: Frontier-Object Maps for Object Goal Navigation
• AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent
• Minimal neuron ablation triggers catastrophic collapse in the language core of Large Vision-Language Models
• BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
• ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction
• When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTI
• ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
• DialBench: Towards Accurate Reading Recognition of Pointer Meter using Large Foundation Models
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
• Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
• From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
• From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
• AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
• Unexplored flaws in multiple-choice VQA evaluations
• UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation
• UNION: A Lightweight Target Representation for Efficient Zero-Shot Image-Guided Retrieval with Optional Textual Queries
• SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
• Co-Training Vision Language Models for Remote Sensing Multi-task Learning
• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
• G²VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
• EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens
• Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI
• AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
• A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control
• VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
• CLASH: A Benchmark for Cross-Modal Contradiction Detection
• BackdoorVLM: A Benchmark for Backdoor Attacks on Vision-Language Models
• Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
• Understanding Task Transfer in Vision-Language Models
• Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
• Exploring Weak-to-Strong Generalization for CLIP-based Classification
• Synthetic Curriculum Reinforces Compositional Text-to-Image Generation
• DiVE-k: Differential Visual Reasoning for Fine-grained Image Recognition
• Can a Second-View Image Be a Language? Geometric and Semantic Cross-Modal Reasoning for X-ray Prohibited Item Detection
• ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization
• VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridging
• Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
• FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning
• MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Use
• Test-Time Temporal Sampling for Efficient MLLM Video Understanding
• When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
• SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation
• Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
• UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
• A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback
• UAM: A Unified Attention-Mamba Backbone of Multimodal Framework for Tumor Cell Classification
• SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion
• Loomis Painter: Reconstructing the Painting Process
• Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
• TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning
• Multimodal Evaluation of Russian-language Architectures
• SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
• Parameter Importance-Driven Continual Learning for Foundation Models
• ChartEditor: A Reinforcement Learning Framework for Robust Chart Editing
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
• Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
• ManipShield: A Unified Framework for Image Manipulation Detection, Localization and Explanation
• Jailbreaking Large Vision Language Models in Intelligent Transportation Systems
• Weakly Supervised Ephemeral Gully Detection In Remote Sensing Images Using Vision Language Models
• Learning Skill-Attributes for Transferable Assessment in Video
• VLMs Guided Interpretable Decision Making for Autonomous Driving
• CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product
• Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization
• Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing
• EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning
• RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
• RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
• Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning
• Suppressing VLM Hallucinations with Spectral Representation Filtering
• PRISM of Opinions: A Persona-Reasoned Multimodal Framework for User-centric Conversational Stance Detection
• TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
• GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving
• Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models
• Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models
• Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque
• RGMP: Recurrent Geometric-prior Multimodal Policy for Generalizable Humanoid Robot Manipulation
• Compression then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
• Remodeling Semantic Relationships in Vision-Language Fine-Tuning
• Multimodal LLMs Do Not Compose Skills Optimally Across Modalities
• NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
• How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation
• A Low-Rank Method for Vision Language Model Hallucination Mitigation in Autonomous Driving
• AesTest: Measuring Aesthetic Intelligence from Perception to Production
• AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
• VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation
• S²LM: Towards Semantic Steganography via Large Language Models
• LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
• Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
• DeepEyesV2: Toward Agentic Multimodal Model
• Visual Spatial Tuning
• Cambrian-S: Towards Spatial Supersensing in Video
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
• V-Thinker: Interactive Thinking with Images
• Embodiment Transfer Learning for Vision-Language-Action Models
• Seeing Straight: Document Orientation Detection for Efficient OCR
• NVIDIA Nemotron Nano V2 VL
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
• Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
• TRACE: Textual Reasoning for Affordance Coordinate Extraction
• When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
• Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis
• TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
• Routing-Based Continual Learning for Multimodal Large Language Models
• A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis
• Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
• Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
• Reimagining Safety Alignment with An Image
• Text-guided Fine-Grained Video Anomaly Understanding
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• Generating Accurate and Detailed Captions for High-Resolution Images
• NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding
• MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models
• Masked Diffusion Captioning for Visual Feature Learning
• ChartAB: A Benchmark for Chart Grounding & Dense Alignment
• SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models
• Emu3.5: Native Multimodal Models are World Learners
• Instance-Level Composed Image Retrieval
• MedVLSynther: Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
• Robotic Assistant: Completing Collaborative Tasks with Dexterous Vision-Language-Action Models
• ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents
• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
• Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
• Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
• InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictions
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
• Can neurons speak? Semantic narration of vision at single-cell resolution
• SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
• Patronus: Safeguarding Text-to-Image Models against White-Box Adversaries
• RL makes MLLMs see better than SFT
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies
• CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding
• Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student
• SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
• OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models
• SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space
• What do vision-language models see in the context? Investigating multimodal in-context learning
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
• Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
• Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
• PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity
• M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
• COOPERA: Continual Open-Ended Human-Robot Assistance
• Large language model-based task planning for service robots: A review
• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
• HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
• FlowOpt: Fast Optimization Through Whole Flow Processes for Training-Free Editing
• FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
• MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection
• Enhanced MLLM Black-Box Jailbreaking Attacks and Defenses
• Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
• SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation
• SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior
• UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis
• ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
• HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
• Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
• Target-aware Image Editing via Cycle-consistent Constraints
• Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation
• MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
• PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning
• Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
• Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
• The Impact of Image Resolution on Biomedical Multimodal Large Language Models
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
• PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions
• Accelerating Vision Transformers with Adaptive Patch Sizes
• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
• Token-Level Inference-Time Alignment for Vision-Language Models
• Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks
• ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion
• SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Driving
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
• SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
• Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?
• Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
• Neuro-Symbolic Spatial Reasoning in Segmentation
• Train a Unified Multimodal Data Quality Classifier with Synthetic Data
• From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
• Benchmarking Multimodal Large Language Models for Face Recognition
• Spatial Preference Rewarding for MLLMs Spatial Understanding
• Vision-Centric Activation and Coordination for Multimodal Large Language Models
• Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding
• Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
• Multimodal Function Vectors for Visual Relations
• InfraGPT Smart Infrastructure: An End-to-End VLM-Based Framework for Detecting and Managing Urban Defects
• Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
• Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment
• Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models
• Reasoning in Space via Grounding in the World
• AVAR-Net: A Lightweight Audio-Visual Anomaly Recognition Framework with a Benchmark Dataset
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
• DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
• CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
• SafeMT: Multi-turn Safety for Multimodal Language Models
• Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model
• ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?
• Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering
• FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
• COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
• A Survey on Agentic Multimodal Large Language Models
• Judge Before Answer: Can MLLM Discern the False Premise in Question?
• Topological Alignment of Shared Vision-Language Embedding Space
• Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
• Learning Dynamics of VLM Finetuning
• Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting
• Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey
• ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency
• Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization
• Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
• CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
• Holistic Order Prediction in Natural Scenes
• Text Prompt Injection of Vision Language Models
• Task-Aware Resolution Optimization for Visual Large Language Models
• Zero-shot image privacy classification with Vision-Language Models
• Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
• UniVideo: Unified Understanding, Generation, and Editing for Videos
• NavSpace: How Navigation Agents Follow Spatial Intelligence Instructions
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
• Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recovered
• RetouchLLM: Training-free Code-based Image Retouching with Vision Language Models
• MLLM4TS: Leveraging Vision and Multimodal Language Models for General Time-Series Analysis
• TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription
• Growing Visual Generative Capacity for Pre-Trained MLLMs
• Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking
• TTRV: Test-Time Reinforcement Learning for Vision Language Models
• StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
• VLA-R1: Enhancing Reasoning in Vision-Language-Action Models
• Get RICH or Die Scaling: Profitably Trading Inference Compute for Robustness
• Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flow
• Vipera: Blending Visual and LLM-Driven Guidance for Systematic Auditing of Text-to-Image Generative AI
• ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations
• Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
• Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models
• Visual Representations inside the Language Model
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
• A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering
• Zephyrus: An Agentic Framework for Weather Science
• AgriGPT-VL: Agricultural Vision-Language Understanding Suite
• FinCall-Surprise: A Large Scale Multi-modal Benchmark for Earning Surprise Prediction
• Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
• Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs
• Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights
• MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
• Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
• ModernVBERT: Towards Smaller Visual Document Retrievers
• Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
• VIRTUE: Visual-Interactive Text-Image Universal Embedder
• TsLLM: Augmenting LLMs for General Time Series Understanding and Prediction
• Can World Models Benefit VLMs for World Dynamics?
• Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
• Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories
• WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents
• From Seeing to Predicting: A Vision-Language Framework for Trajectory Forecasting and Controlled Video Generation
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
• OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
• Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
• MuSLR: Multimodal Symbolic Logical Reasoning
• VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought
• DescribeEarth: Describe Anything for Remote Sensing Images
• Generalized Contrastive Learning for Universal Multimodal Retrieval
• Can AI agents understand spoken conversations about data visualizations in online meetings?
• PRPO: Paragraph-level Policy Optimization for Vision-Language Deepfake Detection
• VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
• FishNet++: Analyzing the capabilities of Multimodal Large Language Models in marine biology
• Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
• Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition
• From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
• Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
• MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech
• SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
• OIG-Bench: A Multi-Agent Annotated Benchmark for Multimodal One-Image Guides Understanding
• Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
• Vision Function Layer in Multimodal LLMs
• Dynamic Orchestration of Multi-Agent System for Real-World Multi-Image Agricultural VQA
• When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs
• Model Correlation Detection via Random Selection Probing
• World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
• VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
• Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation
• HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
• LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
• HIVTP: A Training-Free Method to Improve VLMs Efficiency via Hierarchical Visual Token Pruning Using Middle-Layer-Based Importance Score
• HunyuanImage 3.0 Technical Report
• Uncovering Grounding IDs: How External Cues Shape Multimodal Binding
• RIV: Recursive Introspection Mask Diffusion Vision Language Model
• SPIKE-RL: Video-LLMs meet Bayesian Surprise
• CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
• Culture In a Frame: C³B as a Comic-Based Benchmark for Multimodal Culturally Awareness
• Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
• MMPB: It's Time for Multi-Modal Personalization
• GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting
• Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
• LucidFlux: Caption-Free Universal Image Restoration via a Large-Scale Diffusion Transformer
• Multilingual Vision-Language Models, A Survey
• Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation
• ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
• WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM
• From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
• Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning
• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
• Spatial Reasoning in Foundation Models: Benchmarking Object-Centric Spatial Understanding
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
• MMPlanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning
• Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices
• VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
• Does FLUX Already Know How to Perform Physically Plausible Image Composition?
• FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction
• DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
• ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation
• EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation
• CAMILA: Context-Aware Masking for Image Editing with Language Alignment
• Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models
• LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference
• Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs
• Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers
• Gaze Heads: How VLMs Look at What They Describe
• VCE: A Zero-Cost Hallucination Mitigation Method of LVLMs via Visual Contrastive Editing
• S-GRPO: Unified Post-Training for Large Vision-Language Models
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
• AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
• DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture
• Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
• MAPO: Mixed Advantage Policy Optimization
• COLT: Enhancing Video Large Language Models with Continual Tool Usage
• Knowledge Transfer from Interaction Learning
• RSVG-ZeroOV: Exploring a Training-Free Framework for Zero-Shot Open-Vocabulary Visual Grounding in Remote Sensing Images
• Steering Multimodal Large Language Models Decoding for Context-Aware Safety
• The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographers
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models
• UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
• Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning
• Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• MLLM-Driven Semantic Identifier Generation for Generative Cross-Modal Retrieval
• Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
• Can GRPO Boost Complex Multimodal Table Understanding?
• Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?
• When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
• Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
• CGTGait: Collaborative Graph and Transformer for Gait Emotion Recognition
• Agentic Reasoning for Robust Vision Systems via Increased Test-Time Compute
• Randomized Smoothing Meets Vision-Language Models
• Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
• VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
• Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
• ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models
• Efficient Multimodal Dataset Distillation via Generative Models
• UMind: A Unified Multitask Network for Zero-Shot M/EEG Visual Decoding
• Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
• VLA-LPAF: Lightweight Perspective-Adaptive Fusion for Vision-Language-Action to Enable More Unconstrained Robotic Manipulation
• Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech
• V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
• TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding
• MARIC: Multi-Agent Reasoning for Image Classification
• A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
• AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
• CLAW: A Vision-Language-Action Framework for Weight-Aware Robotic Grasping
• Pre-Manipulation Alignment Prediction with Parallel Deep State-Space and Transformer Models
• Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation
• Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR
• Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
• Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
• Explain Before You Answer: A Survey on Compositional Visual Reasoning
• Image Realness Assessment and Localization with Multimodal Features
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
• The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
• AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
• The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
• 3D Aware Region Prompted Vision Language Model
• AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment
• Cross-Platform Scaling of Vision-Language-Action Models from Edge to Cloud GPUs
• When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models
• Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models
• Pathological Truth Bias in Vision-Language Models
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations
• Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection
• The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge
• Traffic-MLLM: Curiosity-Regularized Supervised Learning for Traffic Scenario Case-Based Reasoning
• Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Exploration
• Towards Understanding Visual Grounding in Visual Language Models
• A Modular and Multimodal Generative AI Framework for Urban Building Energy Data: Generating Synthetic Homes
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization
• BcQLM: Efficient Vision-Language Understanding with Distilled Q-Gated Cross-Modal Fusion
• PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science
• Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning
• TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making
• RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentation
• Recurrence Meets Transformers for Universal Multimodal Retrieval
• Visual Representation Alignment for Multimodal Large Language Models
• TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models
• Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
• Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization
• LLaDA-VLA: Vision Language Diffusion Action Models
• Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
• Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
• Text4Seg++: Advancing Image Segmentation via Generative Language Modeling
• TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detection
• FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction
• Measuring How (Not Just Whether) VLMs Build Common Ground
• MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting
• Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model
• Aesthetic Image Captioning with Saliency Enhanced MLLMs
• Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understanding
• Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens
• OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
• Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
• RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Events
• PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
• Reinforced Visual Perception with Tools
• Improving Large Vision and Language Models by Learning from a Panel of Peers
• POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion
• OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
• OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
• CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
• Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
• TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
• Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
• Waste-Bench: A Comprehensive Benchmark for Evaluating VLLMs in Cluttered Environments
• DriveQA: Passing the Driving Knowledge Test
• CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models
• How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images
• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
• UItron: Foundational GUI Agent with Advanced Perception and Planning
• R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning
• EO-1: Interleaved Vision-Text-Action Pretraining for General Robot Control
• StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
• CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
• Improving Alignment in LVLMs with Debiased Self-Judgment
• MobileCLIP2: Improving Multi-Modal Reinforced Training
• GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
• Video-LLMs with Temporal Visual Screening
• Scalable Object Detection in the Car Interior With Vision Foundation Models
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
• How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
• ZeST: an LLM-based Zero-Shot Traversability Navigation for Unknown Environments
• Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
• Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
• Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum
• Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models
• MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• EcomMMMU: Strategic Utilization of Visuals for Robust Multimodal E-commerce Models
• Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
• STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models
• Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
• Learning to Steer: Input-dependent Steering for Multimodal LLMs
• Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection
• Adaptive Reinforcement for Open-ended Medical Reasoning via Semantic-Guided Reward Collapse Mitigation
• ViDA-UGC: Detailed Image Quality Analysis via Visual Distortion Assessment for UGC Images
• Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
• Simple o3: Towards Interleaved Vision-Language Reasoning
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
• OVG-HQ: Online Video Grounding with Hybrid-modal Queries
• Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems
• Controlling Multimodal LLMs via Reward-guided Decoding
• Language models align with brain regions that represent concepts across modalities
• Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?
• When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
• Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models
• JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics
• MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs
• MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning
• From Intent to Execution: Multimodal Chain-of-Thought Reinforcement Learning for Precise CAD Code Generation
• LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
• On Negative-aware Preference Optimization for Recommendation
• SafeFix: Targeted Model Repair via Controlled Image Generation
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
• Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
• VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
• MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization
• Grounding Multilingual Multimodal LLMs With Cultural Knowledge
• MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
• BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
• Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
• SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding
• The Trauma THOMPSON Dataset for Real-World Emergency AI
• SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning
• AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance
• PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation
• SIFThinker: Spatially-Aware Image Focus for Visual Reasoning
• Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
• LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
• LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model
• VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
• IAD-R1: Reinforcing Consistent Reasoning in Industrial Anomaly Detection
• A Survey on Video Temporal Grounding with Multimodal Large Language Model
• X-SAM: From Segment Anything to Any Segmentation
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
• Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
• ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
• UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval
• Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models
• Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
• IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models
• SCFlow: Implicitly Learning Style and Content Disentanglement with Flow Models
• ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools
• SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
• ChartCap: Mitigating Hallucination of Dense Chart Captioning
• ADSeeker: A Knowledge-Infused Framework for Anomaly Detection and Reasoning
• MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
• Evaluating Variance in Visual Question Answering Benchmarks
• Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
• What are you sinking? A geometric approach on attention sink
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
• A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
• MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
• Advancing the Foundation Model for Music Understanding
• MultiSHAP: A Shapley-Based Framework for Explaining Cross-Modal Interactions in Multimodal AI Models
• HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models
• Fine-grained Spatiotemporal Grounding on Egocentric Videos
• From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model
• CoRGI: Verified Chain-of-Thought Reasoning with Post-hoc Visual Grounding
• Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
• Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models
• Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
• On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
• Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation
• Visual Language Models as Zero-Shot Deepfake Detectors
• What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models
• HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
• Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding
• X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
• MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
• MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces
• Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval
• ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
• Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
• Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Models
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model
• Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
• Enhancing Spatial Reasoning through Visual and Textual Thinking
• AgroBench: Vision-Language Model Benchmark in Agriculture
• Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
• Region-based Cluster Discrimination for Visual Representation Learning

Related

• An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
• LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
• LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
• CROME: Cross-Modal Adapters for Efficient Multimodal LLM
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs