PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratch

2025/10/08 by Shanshan Yin, Su Liang, Yin, Shangjian +10 · 1 citation
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Code (set theory) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Learning curve #Path (computing) #Quality (philosophy) #Reinforcement learning #Scalability #Scratch #Training set

paper · pdf · doi:10.48550/arxiv.2510.06670

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/08 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

High-quality instruction data is critical for LLM alignment, yet existing open-source datasets often lack efficiency, requiring hundreds of thousands of examples to approach proprietary performance. In this work, we find that beyond the widely recognized importance of prompt-response quality, prompt difficulty itself plays a critical role in driving alignment gains. Motivated by this observation, we introduce PiKa, a data-efficient family of expert-level alignment datasets that concentrates supervision on high-difficulty instructions. The PiKa-SFT dataset contains only 30k examples, an order of magnitude fewer than state-of-the-art open datasets like Magpie-Pro. Despite its small size, fine-tuning Llama-3-8B-Base on PiKa-SFT even outperforms the official Llama-3-8B-Instruct model trained on over 10M proprietary examples on widely used benchmarks such as AlpacaEval 2.0 and Arena-Hard. We also validate the generalizability of PiKa across the Qwen2.5 series (0.5B-7B), consistently surpassing their official instruction-tuned counterparts. Additionally, we provide 30k high-quality preference optimization examples to further enhance alignment. Our results demonstrate that promising alignment is achievable with significantly reduced data, democratizing access for resource-constrained research. Our code and data will be available at https://github.com/SJY8460/PiKa.

Citations

• Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
• Scaling Synthetic Data Creation with 1,000,000,000 Personas
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Best Practices and Lessons Learned on Synthetic Data
• CodecLM: Aligning Language Models with Tailored Synthetic Data
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• In-context Autoencoder for Context Compression in a Large Language Model
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• LIMA: Less Is More for Alignment
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• GPT-4 Technical Report
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Finetuned Language Models Are Zero-Shot Learners
• Measuring Massive Multitask Language Understanding
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• UltraFeedback: Boosting Language Models with Scaled AI Feedback

Cited by

• Aligning Large Language Models via Fully Self-Synthetic Data

Related

• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• Aligning Large Language Models through Synthetic Feedback
• Progressively Label Enhancement for Large Language Model Alignment
• SAIL: Self-Improving Efficient Online Alignment of Large Language Models
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment