Human-Instruction-Free LLM Self-Alignment with Limited Samples

2024/01/06 by Hongyi Guo, Guo, Hongyi, Yuanshun Yao +11 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2401.06785

openalex publication_date 2024/01/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.

Cited by

• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• Robust Preference Alignment via Directional Neighborhood Consensus
• Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
• Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
• Not All Documents Are What You Need for Extracting Instruction Tuning Data

Related

• Sample-Efficient Alignment for LLMs
• A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More
• Efficient Alignment of Large Language Models via Data Sampling
• A Survey on Training-free Alignment of Large Language Models
• I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm