Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

2025/08/07 by Nakamura, Mason, Mahmud, Saaduddin, Wray, Kyle H. +2
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #I.2.8 #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.05165

Abstract

Aligning LLMs with user preferences is crucial for real-world use but often requires costly fine-tuning or expensive inference, forcing trade-offs between alignment quality and computational cost. Existing inference-time methods typically ignore this balance, focusing solely on the optimized policy's performance. We propose HIA (Heuristic-Guided Inference-time Alignment), a tuning-free, black-box-compatible approach that uses a lightweight prompt optimizer, heuristic reward models, and two-stage filtering to reduce inference calls while preserving alignment quality. On real-world prompt datasets, HelpSteer and ComPRed, HIA outperforms best-of-N sampling, beam search, and greedy search baselines in multi-objective, goal-conditioned tasks under the same inference budget. We also find that HIA is effective under low-inference budgets with as little as one or two response queries, offering a practical solution for scalable, personalized LLM deployment.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning
• Open Sourcing GPTs: Economics of Open Sourcing Advanced AI Models
• A Comparative Study of Clinical ModernBERT and BioMedical ModernBERT on the DDXPlus Dataset
• Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
• Fast Best-of-N Decoding via Speculative Rejection
• ComPO: Community Preferences for Language Model Personalization
• PAD: Personalized Alignment of LLMs at Decoding-Time
• HelpSteer2-Preference: Complementing Ratings with Preferences
• HelpSteer2: Open-source dataset for training top-performing reward models
• A Survey on Large Language Models for Code Generation
• MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
• HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM
• Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
• PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization
• SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• LaMP: When Large Language Models Meet Personalization
• GPT-4 Technical Report
• Training language models to follow instructions with human feedback
• Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Learning to summarize from human feedback
• Language Models are Few-Shot Learners
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• MaxMin-RLHF: Alignment with Diverse Human Preferences

Related

• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
• MAVIS: Multi-Objective Alignment via Value-Guided Inference-Time Search
• Inference time LLM alignment in single and multidomain preference spectrum
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
• Inference Time Alignment with Reward-Guided Tree Search