ARGS: Alignment as Reward-Guided Search

2024/01/23 by Maxim Khanov, Khanov, Maxim, Jirayu Burapacheep +3 · 38 citations
Business, Management and Accounting · Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Consumer Market Behavior and Pricing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Psychology #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2402.01694

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/01/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Aligning large language models with human objectives is paramount, yet common approaches including RLHF suffer from unstable and resource-intensive training. In response to this challenge, we introduce ARGS, Alignment as Reward-Guided Search, a novel framework that integrates alignment into the decoding process, eliminating the need for expensive RL training. By adjusting the model's probabilistic predictions using a reward signal, ARGS generates texts with semantic diversity while being aligned with human preferences, offering a promising and flexible solution for aligning language models. Notably, ARGS demonstrates consistent enhancements in average reward compared to baselines across diverse alignment tasks and various model dimensions. For example, under the same greedy-based decoding strategy, our method improves the average reward by 19.56% relative to the baseline and secures a preference or tie score of 64.33% in GPT-4 evaluation. We believe that our framework, emphasizing decoding-time alignment, paves the way for more responsive language models in the future. Code is publicly available at: \urlhttps://github.com/deeplearning-wisc/args.

Cited by

• The Reward Model Selection Crisis in Personalized Alignment
• Factors That Support Grounded Responses in LLM Conversations: A Rapid Review
• SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
• Inference-Time Personalized Alignment with a Few User Preference Queries
• Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral
• Adaptive Blockwise Search: Inference-Time Alignment for Large Language Models
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training
• MatryoshkaThinking: Recursive Test-Time Scaling Enables Efficient Reasoning
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts
• Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
• T-POP: Test-Time Personalization with Online Preference Feedback
• IFHierBench: Hierarchical Instruction Following for Large Language Models
• Virtual Agent Economies
• CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention
• Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
• Controlling Multimodal LLMs via Reward-guided Decoding
• A Survey on Training-free Alignment of Large Language Models
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
• LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation
• Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
• Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
• When Can Model-Free Reinforcement Learning be Enough for Thinking?
• Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples
• Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
• From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment
• Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling
• Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignment
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
• BiasFilter: An Inference-Time Debiasing Framework for Large Language Models
• Multi-objective Large Language Model Alignment with Hierarchical Experts
• IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios
• Temporal Sampling for Forgotten Reasoning in LLMs
• Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment
• Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
• MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO
• AdaBoN: Adaptive Best-of-N Alignment
• CEC-Zero: Chinese Error Correction Solution Based on LLM
• TTRL: Test-Time Reinforcement Learning
• Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
• A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

Related

• DeAL: Decoding-time Alignment for Large Language Models
• Alignment-Aware Decoding
• Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
• Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
• Energy-Based Reward Models for Robust Language Model Alignment