Fine-Tuning Language Models from Human Preferences

2019/09/18 by Daniel M. Ziegler, Ziegler, Daniel M., Nisan Stiennon +15 · 481 citations
Computer Science · Mathematics · #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1909.08593

arxiv created 2020/01/08 · arxiv updated 2020/01/10

Abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

Citations

• Semi-supervised Sequence Learning
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning to Generate Reviews and Discovering Sentiment
• Supervising strong learners by amplifying weak experts
• Scalable agent alignment via reward modeling: a research direction

Cited by

• Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
• Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
• Trust Region Masking for Long-Horizon LLM Reinforcement Learning
• Towards Understanding Steering Strength
• ReDiF: Reinforced Distillation for Few Step Diffusion
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• What do Reward Models Memorize?
• Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
• Group Preference Collapse in Personalized Multimodal Large Language Models
• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
• A Comedy of Estimators: On KL Regularization in RL Training of LLMs
• Beyond Context: Large Language Models' Failure to Grasp Users' Intent
• ORPR: An OR-Guided Pretrain-then-Reinforce Learning Model for Inventory Management
• ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
• Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
• Offline Multi-Task Multi-Objective Data-Driven Evolutionary Algorithm with Language Surrogate Model and Implicit Q-Learning
• Georeferencing complex relative locality descriptions with large language models
• Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries
• Explainable reinforcement learning from human feedback to improve alignment
• CAPE: Capability Achievement via Policy Execution
• AIR: Post-training Data Selection for Reasoning via Attention Head Influence
• Unified Control for Inference-Time Guidance of Denoising Diffusion Models
• Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization
• Intelligently Weighting Multiple Reference Models for Direct Preference Optimization of LLMs
• Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
• Gradient-Informed Monte Carlo Fine-Tuning of Diffusion Models for Low-Thrust Trajectory Design
• Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
• When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
• CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography
• On The Finetuning of MLIPs Through the Lens of Iterated Maps With BPTT
• When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF
• Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
• Learning the MPC objective function from human preferences
• Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO
• Benchmarking In-context Experiential Learning Through Repeated Product Recommendations
• Bootstrapping LLMs via Preference-Based Policy Optimization
• Steering Generative Models for Protein Design: Aligning and Conditioning Strategies
• The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training
• GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs
• Reinforcement Learning for Latent-Space Thinking in LLMs
• OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
• Learning to Clean: Reinforcement Learning for Noisy Label Correction
• Why Do Language Model Agents Whistleblow?
• Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective
• Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models
• TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
• Distribution Matching Distillation Meets Reinforcement Learning
• RLHF May Not Reflect Genuine Preferences
• Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• Maximizing the efficiency of human feedback in AI alignment: a comparative analysis
• Mitigating Length Bias in RLHF through a Causal Lens
• Hierarchical Memorization in Large Language Models: Evidence from Citation Generation
• RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic Transformation
• DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF
• HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection
• Learning Norms from Stories: A Prior for Value Aligned Agents
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• Too Good to be Bad: On the Failure of LLMs to Role-Play Villains
• Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference
• Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
• Inference-Time Personalized Alignment with a Few User Preference Queries
• RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks
• Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events?
• MolChord: Structure-Sequence Alignment for Protein-Guided Drug Design
• Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral
• Test-Time Alignment of LLMs via Sampling-Based Optimal Control in pre-logit space
• Approximating Human Preferences Using a Multi-Judge Learned System
• Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy
• Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments
• NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO
• Training Conversational Agents with Generative Conversational Networks
• Generating Self-Contained and Summary-Centric Question Answer Pairs via Differentiable Reward Imitation Learning
• Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
• Developing Students’ Statistical Expertise Through Writing in the Age of AI
• GeDi: Generative Discriminator Guided Sequence Generation
• Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
• Normatively guided graph diffusion for efficient and rational utility generation in bilateral bargaining
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
• Greedy Sampling Is Provably Efficient for RLHF
• Semi-Supervised Preference Optimization with Limited Feedback
• The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
• Towards Scalable Oversight via Partitioned Human Supervision
• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
• Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
• Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
• Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
• Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
• Robust Preference Alignment via Directional Neighborhood Consensus
• Ask a Strong LLM Judge when Your Reward Model is Uncertain
• Why DPO is a Misspecified Estimator and How to Fix It
• Deep Extrapolation for Attribute-Enhanced Generation
• Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
• DP²O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution
• Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Eliciting Truthful Feedback for Preference-Based Learning via the VCG Mechanism
• Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
• Reinforcement Learning for Unsupervised Domain Adaptation in Spatio-Temporal Echocardiography Segmentation
• Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment
• From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• Repairing Reward Functions with Human Feedback to Mitigate Reward Hacking
• On the Role of Preference Variance in Preference Optimization
• A Survey on Evaluation of Large Language Models
• How Well Can Preference Optimization Generalize Under Noisy Feedback?
• Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
• Reinforced Preference Optimization for Recommendation
• MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
• MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• A-IPO: Adaptive Intent-driven Preference Optimization
• Users as Annotators: LLM Preference Learning from Comparison Mode
• Opponent Shaping in LLM Agents
• Energy-Driven Steering: Reducing False Refusals in Large Language Models
• On the optimization dynamics of RLVR: Gradient gap and step size thresholds
• MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
• Reinforcing Diffusion Models by Direct Group Preference Optimization
• Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
• Textual interpretation of transient image classifications from large language models
• Aligning Large Language Models via Fully Self-Synthetic Data
• Towards Better Optimization For Listwise Preference in Diffusion Models
• LLM Bias Detection and Mitigation through the Lens of Desired Distributions
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
• LLMs as Policy-Agnostic Teammates: A Case Study in Human Proxy Design for Heterogeneous Agent Teams
• Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
• Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
• From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models
• TeachLM: Post-Training LLMs for Education Using Authentic Learning Data
• DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
• Self-supervised diffusion model fine-tuning for costate initialization using Markov chain Monte Carlo
• Best of mini-N in-loop Sampling: A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• A Qualitative Comparative Evaluation of Cognitive and Generative Theories
• GEM: A Gym for Agentic LLMs
• Rethinking Reward Models for Multi-Domain Test-Time Scaling
• Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
• Alignment-Aware Decoding
• Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
• Probing the Limits of Stylistic Alignment in Vision-Language Models
• SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
• The Era of Real-World Human Interaction: RL from User Conversations
• T-POP: Test-Time Personalization with Online Preference Feedback
• Humanline: Online Alignment as Perceptual Loss
• RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
• Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback
• Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
• Reinforcement Mid-Training
• Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training
• Anchored Supervised Fine-Tuning
• Why Alignment Must Precede Distillation: A Minimal Working Explanation
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• Tree Reward-Aligned Search for TReASURe in Masked Diffusion Language Models
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
• Fine-tuning of Large Language Models for Domain-Specific Cybersecurity Knowledge
• PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs
• Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
• TAPO: Transition-Aware Policy Optimization for LLM Agents
• Conducting Qualitative Interviews with AI
• Proximal Supervised Fine-Tuning
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
• Explore the Reinforcement Learning for the LLM based ASR and TTS system
• Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports
• Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization
• RelRepair: Enhancing Automated Program Repair by Retrieving Relevant Code
• The Alignment Bottleneck
• Risk Assessment and Security Analysis of Large Language Models
• A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
• REFINER: Reasoning Feedback on Intermediate Representations
• Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
• Recursively Summarizing Books with Human Feedback
• Designing Rules to Pick a Rule: Aggregation by Consistency
• Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization
• MusicSwarm: Biologically Inspired Intelligence for Music Composition
• Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents
• KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF
• Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration
• Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization
• LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination
• RewardDance: Reward Scaling in Visual Generation
• Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing
• Interpretability as Alignment: Making Internal Understanding a Design Principle
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models
• MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
• Let's Roleplay: Examining LLM Alignment in Collaborative Dialogues
• Post-training Large Language Models for Diverse High-Quality Responses
• Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoT
• Symbolic Graphics Programming with Large Language Models
• Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignment
• Towards a Unified View of Large Language Model Post-Training
• Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
• Language Models are Few-Shot Learners
• Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
• Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
• Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
• Improving Large Vision and Language Models by Learning from a Panel of Peers
• The Need for Verification in AI-Driven Scientific Discovery
• Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
• Reinforcement Learning for Machine Learning Engineering Agents
• RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning
• Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
• Transforming Agency. On the mode of existence of Large Language Models
• Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation
• FormaRL: Enhancing Autoformalization with no Labeled Data
• Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation
• AI Testing Should Account for Sophisticated Strategic Behaviour
• DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
• Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes
• Reinforced Language Models for Sequential Decision Making
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
• Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization
• \(X\)-evolve: Solution space evolution powered by large language models
• Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• The Fair Game: Auditing & Debiasing AI Algorithms Over Time
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs
• Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models
• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
• Sotopia-RL: Reward Design for Social Intelligence
• Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling
• V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Models
• Machine culture
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
• Censored Sampling for Topology Design: Guiding Diffusion with Human Preferences
• A Note on Code Quality Score: LLMs for Maintainable Large Codebases
• Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English
• ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning
• Improving Generative Ad Text on Facebook using Reinforcement Learning
• CTR-Driven Ad Text Generation via Online Feedback Preference Optimization
• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
• Justifications for Democratizing AI Alignment and Their Prospects
• Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
• Checklists Are Better Than Reward Models For Aligning Language Models
• Language Model Evaluation in Open-ended Text Generation
• V₁: Unifying Generation and Self-Verification for Parallel Reasoners
• URPO: A Unified Reward & Policy Optimization Framework for Large Language Models
• Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice
• On a few pitfalls in KL divergence gradient estimation for RL
• Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
• A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges
• Why Isn't Relational Learning Taking Over the World?
• Learning to summarize user information for personalized reinforcement learning from human feedback
• Minimalist Concept Erasure in Generative Models
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
• A Survey on Large Language Models for Mathematical Reasoning
• Repairing Language Model Pipelines by Meta Self-Refining Competing Constraints at Runtime
• Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
• Enhancing RLHF with Human Gaze Modeling
• Pushing the Boundaries of Immersion and Storytelling: A Technical Review of Unreal Engine
• Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
• Why is Your Language Model a Poor Implicit Reward Model?
• Data-Balanced Curriculum Learning for Audio Question Answering
• Large Language Model for Extracting Complex Contract Information in Industrial Scenes
• Differentiable Reward Optimization for LLM based TTS system
• Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests
• The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains
• Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
• MolFORM: Multi-modal Flow Matching for Structure-Based Drug Design
• Energy-Based Models for Code Generation under Compilability Constraints
• Improving Consistency in Vehicle Trajectory Prediction Through Preference Optimization
• Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting
• Activation Reward Models for Few-Shot Model Alignment
• Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
• Scaling Human Judgment in Community Notes with LLMs
• A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
• QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
• A Distributional Approach to Controlled Text Generation
• Treatment, evidence, imitation, and chat
• Generating Privacy Stories From Software Documentation
• Video Unlearning via Low-Rank Refusal Vector
• A Survey of Human-in-the-loop for Machine Learning
• Explicit Preference Optimization: No Need for an Implicit Reward Model
• Aligning Spoken Dialogue Models from User Interactions
• Bridging Offline and Online Reinforcement Learning for LLMs
• Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models
• Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
• Probing AI Safety with Source Code
• ReDit: Reward Dithering for Improved LLM Policy Optimization
• Generalizing vision-language models to novel domains: A comprehensive survey
• Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)
• Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples
• Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability
• Intelligent Assistants for the Semiconductor Failure Analysis with LLM-Based Planning Agents
• HeurAgenix: Leveraging LLMs for Solving Complex Combinatorial Optimization Challenges
• TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
• Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
• From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs
• Fake it till You Make it: Reward Modeling as Discriminative Prediction
• TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence
• Human-assisted Robotic Policy Refinement via Action Preference Optimization
• OneRec Technical Report
• Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning
• Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
• History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM
• Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
• From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment
• Configurable Preference Tuning with Rubric-Guided Synthetic Data
• AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization
• EQA-RM: A Generative Embodied Reward Model with Test-time Scaling
• EnerBridge-DPO: Energy-Guided Protein Inverse Folding with Markov Bridges and Direct Preference Optimization
• Debiasing Online Preference Learning via Preference Feature Preservation
• Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
• SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat
• Selective Matching Losses -- Not All Scores Are Created Equal
• MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
• Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs
• Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach
• Leveraging Reward Models for Guiding Code Review Comment Generation
• Policy-labeled Preference Learning: Is Preference Enough for RLHF?
• Robust Preference Optimization via Dynamic Target Margins
• Misalignment or misuse? The AGI alignment tradeoff
• RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates
• Aligning Large Language Models with Implicit Preferences from User-Generated Content
• RewardAnything: Generalizable Principle-Following Reward Models
• Geospatial Mechanistic Interpretability of Large Language Models
• FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
• Seeing the Arrow of Time in Large Multimodal Models
• DPO Learning with LLMs-Judge Signal for Computer Use Agents
• What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Context
• Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
• ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding
• Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification
• Large language models can learn and generalize steganographic chain-of-thought under process supervision
• A Descriptive and Normative Theory of Human Beliefs in RLHF
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding
• Beyond RLHF: A Unified Theoretical Framework of Alignment
• SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
• HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization
• Doubly Robust Alignment for Large Language Models
• MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning
• QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
• Adversarial Preference Learning for Robust LLM Alignment
• A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming
• Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective
• MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
• MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
• Learning Parametric Distributions from Samples and Preferences
• Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarios
• Accelerating RLHF Training with Reward Variance Increase
• LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
• Preference Learning with Response Time: Robust Losses and Guarantees
• Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy
• Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R²)GRPO
• Learning Natural Language Generation from Scratch
• Out of the Past: An AI-Enabled Pipeline for Traffic Simulation from Noisy, Multimodal Detector Data and Stakeholder Feedback
• Reinforcing General Reasoning without Verifiers
• LPOI: Listwise Preference Optimization for Vision Language Models
• Multi-objective Large Language Model Alignment with Hierarchical Experts
• SquareχPO: Differentially Private and Robust χ²-Preference Optimization in Offline Direct Alignment
• RM-R1: Reward Modeling as Reasoning
• Aligning LLMs by Predicting Preferences from User Writing Samples
• Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
• Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
• Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs
• Risk-aware Direct Preference Optimization under Nested Risk Measure
• Preference Optimization by Estimating the Ratio of the Data Distribution
• Learning to Reason without External Rewards
• Learning to Select In-Context Demonstration Preferred by Large Language Model
• Deep Actor-Critics with Tight Risk Certificates
• FairPO: Robust Preference Optimization for Fair Multi-Label Learning
• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
• Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Training
• Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers
• A Survey on Progress in LLM Alignment from the Perspective of Reward Design
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• Incentivizing High-Quality Human Annotations with Golden Questions
• Online Knowledge Distillation with Reward Guidance
• GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chains
• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
• OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
• Reward Model Overoptimisation in Iterated RLHF
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Calibrate your listeners! Robust communication-based training for pragmatic speakers
• KL-regularization Itself is Differentially Private in Bandits and RLHF
• MPO: Multilingual Safety Alignment via Reward Gap Optimization
• Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator
• A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
• MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
• Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactions
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
• Self-Evolving Curriculum for LLM Reasoning
• Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
• Reinforcement Learning from User Feedback
• DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
• Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
• Neural Language Generation: Formulation, Methods, and Evaluation
• UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
• DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
• ExpertSteer: Intervening in LLMs through Expert Knowledge
• SafeVid: Toward Safety Aligned Video Large Multimodal Models
• Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity
• OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
• Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
• A Systematic Analysis of Base Model Choice for Reward Modeling
• Group-in-Group Policy Optimization for LLM Agent Training
• Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations
• ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization
• Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approach
• Fast Text-to-Audio Generation with Adversarial Post-Training
• Freeform Preference Learning for Robotic Manipulation
• Improved Algorithms for Differentially Private Language Model Alignment
• On the Robustness of Reward Models for Language Model Alignment
• Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
• You Only Look One Step: Accelerating Backpropagation in Diffusion Sampling with Gradient Shortcuts
• Evolutionary thoughts: integration of large language models and evolutionary algorithms
• SOD: Step-wise On-policy Distillation for Small Language Model Agents
• From OSS to Open Source AI: an Exploratory Study of Collaborative Development Paradigm Divergence
• ComPO: Preference Alignment via Comparison Oracles
• Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs
• Semantic Probabilistic Control of Language Models
• Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
• Always Tell Me The Odds: Fine-grained Conditional Probability Estimation
• LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures
• Quo Vadis, World Modeling?
• Constrained Text Generation with Global Guidance -- Case Study on CommonGen
• π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
• Capabilities Ain't All You Need: Measuring Propensities in AI
• Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
• Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation
• Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
• Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets
• Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
• GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets
• Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
• SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
• ThinkFL: Self-Refining Failure Localization for Microservice Systems via Reinforcement Fine-Tuning
• Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization
• Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
• Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology
• SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
• A survey on multimodal large language models
• SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
• Closed-Form Optimal Stepsizes for Cyclic Continual Learning: The Complete d = 2 Theory and the Algebraic Boundary at K = 3
• GeoReward: Mitigating Contextual Variable Overestimation in Vision-Language Models for Cross-Market Preference Prediction
• Document Optimization for Black-Box Retrieval via Reinforcement Learning
• Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
• Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning
• LoRe: Personalizing LLMs via Low-Rank Reward Modeling
• Direct Advantage Regression: Aligning LLMs with Online AI Reward
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
• Evaluating large language models on a highly-specialized topic, radiation oncology physics
• Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo
• Energy-Based Reward Models for Robust Language Model Alignment
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• Evaluating the Diversity and Quality of LLM Generated Content
• BenchQC -- Scalable and modular benchmarking of industrial quantum computing applications
• Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project
• FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
• Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
• Fine-tuning a Large Language Model for Automating Computational Fluid Dynamics Simulations
• SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding
• Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
• SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Data
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling

Related

• Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• Reward Modeling with Weak Supervision for Language Models
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries