TaP: A Taxonomy-Guided Framework for Automated and Scalable Preference Data Generation

2025/06/30 by Renren Jin, Jin, Renren, Tianhao Shen +19
Computer Science · #Machine Learning and Data Classification #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2506.23979

Abstract

Conducting supervised and preference fine-tuning of large language models (LLMs) requires high-quality datasets to improve their ability to follow instructions and align with human preferences and values. However, constructing such datasets is resource-intensive, and most publicly available datasets are in English. To address these challenges, we propose the \underlineTaxonomy-Guided \underlinePreference Data Generation (TaP) framework for automated, scalable preference dataset construction across languages. TaP uses a structured taxonomy to provide fine-grained control over dataset composition, ensuring diversity and broad coverage. We use TaP-generated datasets to perform supervised and preference fine-tuning on multiple LLMs. Experimental results demonstrate that LLMs trained on TaP-generated datasets outperform those trained on existing open-source datasets. Remarkably, LLMs trained on TaP-generated datasets outperform models trained on an open-source dataset that is 180× larger.

Citations

• A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
• From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding
• The Best Instruction-Tuning Data are Those That Fit
• Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
• A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• HelpSteer2: Open-source dataset for training top-performing reward models
• MAmmoTH2: Scaling Instructions from the Web
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
• COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
• HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM
• The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Human Feedback is not Gold Standard
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Textbooks Are All You Need II: phi-1.5 technical report
• Large Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
• Large Language Models are not Fair Evaluators
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• LIMA: Less Is More for Alignment
• WizardLM: Empowering large pre-trained language models to follow complex instructions
• Chinese Open Instruction Generalist: A Preliminary Release
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback

Related

• Multi-Reference Preference Optimization for Large Language Models
• Towards a Unified View of Preference Learning for Large Language Models: A Survey
• RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
• Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
• Investigating on RLHF methodology