RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness

2024/05/27 by Tianyu Yu, Yu, Tianyu, Haoye Zhang +24 · 23 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2405.17220

openalex publication_date 2024/05/27 · openalex created_date 2024/05/29 · openalex updated_date 2026/07/28

Abstract

Traditional feedback learning for hallucination reduction relies on labor-intensive manual labeling or expensive proprietary models. This leaves the community without foundational knowledge about how to build high-quality feedback with open-source MLLMs. In this work, we introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm. RLAIF-V maximally explores open-source MLLMs from two perspectives, including high-quality feedback data generation for preference learning and self-feedback guidance for inference-time scaling. Extensive experiments on six benchmarks in both automatic and human evaluation show that RLAIF-V substantially enhances the trustworthiness of models at both preference learning and inference time. RLAIF-V 7B reduces object hallucination by 80.7% and overall hallucination by 33.7%. Remarkably, RLAIF-V 12B further reveals the self-alignment potential of open-source MLLMs, where the model can learn from feedback of itself to achieve super GPT-4V trustworthiness.

Cited by

• StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
• CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?
• Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
• VisualActBench: Can VLMs See and Act like a Human?
• Self-Improving VLM Judges Without Human Annotations
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
• EcoAlign: An Economically Rational Framework for Efficient LVLM Alignment
• PreResQ-R1: Towards Fine-Grained Rank-and-Score Reinforcement Learning for Visual Quality Assessment via Preference-Response Disentangled Policy Optimization
• RL makes MLLMs see better than SFT
• ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
• DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
• Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
• Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
• SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
• Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
• Mitigating Object Hallucinations via Sentence-Level Early Intervention

Related

• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
• On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
• Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation