Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

2025/05/26 by Yang Zhang, Yu Yu, Zhang, Yang +18
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.19743

openalex publication_date 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.

Citations

• Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
• DPO Meets PPO: Reinforced Token Optimization for RLHF
• From r to Q^*: Your Language Model is Secretly a Q-Function
• Token-level Direct Preference Optimization
• Foundational Challenges in Assuring Alignment and Safety of Large Language Models
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Aligner: Efficient Alignment by Learning to Correct
• Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
• RAIN: Your Language Models Can Align Themselves without Finetuning
• Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears
• Discovering Language Model Behaviors with Model-Written Evaluations
• Affective Coherence Monitoring for Transformer-Based Language Models
• Self-critiquing models for assisting human evaluators
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• WebGPT: Browser-assisted question-answering with human feedback
• Recursively Summarizing Books with Human Feedback
• Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning
• Learning to summarize from human feedback
• Fine-Tuning Language Models from Human Preferences
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Aligning Large Language Models with Human: A Survey

Related

• STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models
• Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
• AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
• Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
• Xwin-LM: Strong and Scalable Alignment Practice for LLMs