Icon²: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

2025/09/06 by Qiyuan Chen, Chen, Qiyuan, Hongsen Huang +15
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.05605

openalex publication_date 2025/09/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) require high quality preference datasets to align with human preferences. However, conventional methods for constructing such datasets face significant challenges: reliance on pre-collected instructions often leads to distribution mismatches with target models, while the need for sampling multiple stochastic responses introduces substantial computational overhead. In this work, we explore a paradigm shift by leveraging inherent regulation of LLMs' representation space for efficient and tailored preference dataset construction, named Icon². Specifically, it first extracts layer-wise direction vectors to encode sophisticated human preferences and then uses these vectors to filter self-synthesized instructions based on their inherent consistency. During decoding, bidirectional inherent control is applied to steer token representations, enabling the precise generation of response pairs with clear alignment distinctions. Experimental results demonstrate significant improvements in both alignment and efficiency. Llama3-8B and Qwen2-7B achieve an average win rate improvement of 13.89% on AlpacaEval 2.0 and 13.45% on Arena-Hard, while reducing computational costs by up to 48.1%.

Citations

• H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
• Self-Boosting Large Language Models with Synthetic Preference Data
• Language Models Learn to Mislead Humans via RLHF
• Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
• Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal
• TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space
• Large Language Models for Data Annotation and Synthesis: A Survey
• Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
• Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
• Aligner: Efficient Alignment by Learning to Correct
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• Aligning Large Language Models with Human Preferences through Representation Engineering
• Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
• Representation Engineering: A Top-Down Approach to AI Transparency
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Discovering Language Model Behaviors with Model-Written Evaluations
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• WebGPT: Browser-assisted question-answering with human feedback
• Ethical and social risks of harm from Language Models
• Identifiability in inverse reinforcement learning
• Language (Technology) is Power: A Critical Survey of "Bias" in NLP
• Language Models are Few-Shot Learners
• CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control
• Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
• UltraFeedback: Boosting Language Models with Scaled AI Feedback

Related

• Self-Boosting Large Language Models with Synthetic Preference Data
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
• Toward Preference-aligned Large Language Models via Residual-based Model Steering
• Parameter-Efficient Tuning Helps Language Model Alignment