Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
2023/12/28 by Haoning Wu, Wu, Haoning, Zicheng Zhang +25 · 135 citations
Computer Science · #Multimodal Machine Learning Applications #Advanced Image and Video Retrieval Techniques #Visual Attention and Saliency Detection
paper · pdf · doi:10.48550/arxiv.2312.17090
Abstract
The explosion of visual content available online underscores the requirement for an accurate machine assessor to robustly evaluate scores across diverse types of visual contents. While recent studies have demonstrated the exceptional potentials of large multi-modality models (LMMs) on a wide range of related fields, in this work, we explore how to teach them for visual rating aligned with human opinions. Observing that human raters only learn and judge discrete text-defined levels in subjective studies, we propose to emulate this subjective process and teach LMMs with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA), as well as video quality assessment (VQA) tasks under the original LMM structure. With the syllabus, we further unify the three tasks into one model, termed the OneAlign. In our experiments, we demonstrate the advantage of the discrete-level-based syllabus over direct-score-based variants for LMMs. Our code and the pre-trained weights are released at https://github.com/Q-Future/Q-Align.
Cited by
- LongCat-Image Technical Report
- IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story Generation
- Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment
- Plug In, Grade Right: Psychology-Inspired AGIQA
- Bridging Your Imagination with Audio-Video Generation via a Unified Director
- AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars
- I2VShield: An Efficient Proactive Defense Framework against DiT-based Image-to-Video Models
- Noise-Free One-Step LoRA for Task-Driven Image Restoration with Diffusion Priors
- StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
- UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture
- DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation
- ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars
- FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction
- Guiding Perception-Reasoning Closer to Human in Blind Image Quality Assessment
- TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation
- JoyAvatar: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
- Zero-shot Adaptation of Stable Diffusion via Plug-in Hierarchical Degradation Representation for Real-World Super-Resolution
- Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment
- Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessment
- WonderZoom: Multi-Scale 3D World Generation
- Unified Camera Positional Encoding for Controlled Video Generation
- UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement
- SA-IQA: Redefining Image Quality Assessment for Spatial Aesthetics with Multi-Dimensional Rewards
- Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
- I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models
- Rethinking Surgical Smoke: A Smoke-Type-Aware Laparoscopic Video Desmoking Method and Dataset
- TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance
- DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategy
- PhotoFramer: Multi-modal Image Composition Instruction
- PanFlow: Decoupled Motion Control for Panoramic Video Generation
- IRPO: Boosting Image Restoration via Post-training GRPO
- Q-Save: Towards Scoring and Attribution for Generated Video Evaluation
- Test-Time Preference Optimization for Image Restoration
- Nested Unfolding Network for Real-World Concealed Object Segmentation
- IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignment
- UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios
- VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment
- Learning Latent Transmission and Glare Maps for Lens Veiling Glare Removal
- Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content
- SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Design
- Distribution Matching Distillation Meets Reinforcement Learning
- Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward
- Co-Layout: LLM-driven Co-optimization for Interior Layout
- Top2Ground: A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation
- Revisiting MLLM Based Image Quality Assessment: Errors and Remedy
- CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video
- MUGSQA: Novel Multi-Uncertainty-Based Gaussian Splatting Quality Assessment Method, Dataset, and Benchmarks
- LMM-IQA: Image Quality Assessment for Low-Dose CT Imaging
- PreResQ-R1: Towards Fine-Grained Rank-and-Score Reinforcement Learning for Visual Quality Assessment via Preference-Response Disentangled Policy Optimization
- Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start
- Evaluating Video Quality Metrics for Neural and Traditional Codecs using 4K/UHD-1 Videos
- ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation
- Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
- LoCoT2V-Bench: A Benchmark for Long-Form and Complex Text-to-Video Generation
- Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
- SpatialQ: Understanding 3D Gaussian Splatting Scene Quality via Visual-based MLLM
- Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration
- DP2O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution
- iDETEX: Empowering MLLMs for Intelligent DETailed EXplainable IQA
- Cost Savings from Automatic Quality Assessment of Generated Images
- ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraints
- FlashWorld: High-quality 3D Scene Generation within Seconds
- DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training
- Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment
- OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment
- DREAM: A Benchmark Study for Deepfake REalism AssessMent
- HandEval: Taking the First Step Towards Hand Quality Evaluation in Generated Images
- HeadsUp! High-Fidelity Portrait Image Super-Resolution
- UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution
- PIT-QMM: A Large Multimodal Model For No-Reference Point Cloud Quality Assessment
- MorphoSim: An Interactive, Controllable, and Editable Language-guided 4D World Simulator
- DHQA-4D: Perceptual Quality Assessment of Dynamic 4D Digital Human
- AgenticIQA: An Agentic Framework for Adaptive and Interpretable Image Quality Assessment
- Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Ranking
- NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding
- VideoScore2: Think before You Score in Generative Video Evaluation
- LucidFlux: Caption-Free Universal Image Restoration via a Large-Scale Diffusion Transformer
- Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
- InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditions
- TuningIQA: Fine-Grained Blind Image Quality Assessment for Livestreaming Camera Tuning
- TinySR: Pruning Diffusion for Real-World Image Super-Resolution
- MVQA-68K: A Multi-dimensional and Causally-annotated Dataset with Quality Interpretability for Video Assessment
- VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results
- Time-Aware One Step Diffusion Network for Real-World Image Super-Resolution
- VQualA 2025 Challenge on Engagement Prediction for Short Videos: Methods and Results
- Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation
- Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA
- Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding
- InfinityHuman: Towards Long-Term Audio-Driven Human
- Style4D-Bench: A Benchmark Suite for 4D Stylization
- OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation
- HiRQA: Hierarchical Ranking and Quality Alignment for Opinion-Unaware Image Quality Assessment
- Fine-grained Image Quality Assessment for Perceptual Image Restoration
- CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance
- Creative4U: MLLMs-based Advertising Creative Image Selector with Comparative Reasoning
- Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models
- Demystifying Foreground-Background Memorization in Diffusion Models
- FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation
- Better Supervised Fine-tuning for VQA: Integer-Only Loss
- A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation
- Images Speak Louder Than Scores: Failure Mode Escape for Enhancing Generative Quality
- Subjective and Objective Quality Assessment of Banding Artifacts on Compressed Videos
- Segmenting and Understanding: Region-aware Semantic Attention for Fine-grained Image Quality Assessment with Large Language Models
- StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
- 3DGS-VBench: A Comprehensive Video Quality Evaluation Benchmark for 3DGS Compression
- VQAThinker: Exploring Generalizable and Explainable Video Quality Assessment via Reinforcement Learning
- PoseGen: In-Context LoRA Finetuning for Pose-Controllable Long Human Video Generation
- AU-IQA: A Benchmark Dataset for Perceptual Quality Assessment of AI-Enhanced User-Generated Content
- StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback
- Refine-IQA: Multi-Stage Reinforcement Finetuning for Perceptual Image Quality Assessment
- Engagement Prediction of Short Videos with Large Multimodal Models
- Efficient Face Image Quality Assessment via Self-training and Knowledge Distillation
- Exploiting Diffusion Prior for Task-driven Image Restoration
- HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
- Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
- Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training
- GT-Mean Loss: A Simple Yet Effective Solution for Brightness Mismatch in Low-Light Image Enhancement
- Position: Reasoning After Perception Means Reasoning Without Vision
- Language Integration in Fine-Tuning Multimodal Large Language Models for Image-Based Regression
- ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding
- Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution
- AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation
- DeQA-Doc: Adapting DeQA-Score to Document Image Quality Assessment
- Elevating 3D Models: High-Quality Texture and Geometry Refinement from a Low-Quality Model
- CharaConsist: Fine-Grained Consistent Character Generation
- NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation
- Latent Posterior-Mean Rectified Flow for Higher-Fidelity Perceptual Face Restoration
- WordCon: Word-level Typography Control in Scene Text Rendering
- Bridging Video Quality Scoring and Justification via Large Multimodal Models
- DreamAnywhere: Object-Centric Panoramic 3D Scene Generation
- WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration
- OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation
- VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning
- MetaQAP - A Meta-Learning Approach for Quality-Aware Pretraining in Image Quality Assessment
- ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies
Related