BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences

2026/07/29 by Ru Peng, Haokai Xu, Xijun Gu +11
Computer Science · #cs.CL

paper · pdf

arxiv created 2026/07/29 · arxiv updated 2026/07/31

Abstract

While data synthesis for large language models (LLMs) is prevalent, it primarily targets domains with verifiable answers, overlooking open-ended humanities and social sciences (HSS), where nuanced quality judgments matter more than objective correctness. This makes preference alignment a natural paradigm for broad HSS tasks. Yet existing methods are either costly or not tailored to broad HSS disciplines. We thus propose BridgeAlign, among the first preference-alignment pipelines for broad HSS disciplines, with three phases: i) Seed Curation: curating HSS seed documents from web corpora via heuristic/LLM-based filtering and text refinement; ii) Preference Data Synthesis: generating preference triplets via persona-based instruction inversion with Q&A consistency checks; iii) Preference Optimization: moving beyond naive human-vs-model heuristics by first grounding preferences in HSS quality rubric, then generating transitional responses via controlled quality degradation to form near-boundary preference pairs for finer-grained quality discrimination. Aligning over 210k synthetic preference samples, BridgeAlign enables Qwen3-8B to achieve the best average across 17 benchmarks against 11 strong baselines; importantly, leading on both human-preference and knowledge-based capabilities at once, with no trade-off between them, as supported by extensive experiments and contextualized by existing theories.

Citations

• From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding
• Qwen3 Technical Report
• Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Prompts
• Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization
• Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
• Preference Leakage: A Contamination Problem in LLM-as-a-judge
• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
• The Llama 3 Herd of Models
• β-DPO: Direct Preference Optimization with Dynamic β
• Scaling Synthetic Data Creation with 1,000,000,000 Personas
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• MAmmoTH2: Scaling Instructions from the Web
• CodecLM: Aligning Language Models with Tailored Synthetic Data
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation
• Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
• LiPO: Listwise Preference Optimization through Learning-to-Rank
• Weaver: Foundation Models for Creative Writing
• Zephyr: Direct Distillation of LM Alignment
• TabuLa: Harnessing Language Models for Tabular Data Synthesis
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• COLLIE: Systematic Construction of Constrained Text Generation Tasks
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Large Language Models as Tool Makers
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• SCROLLS: Standardized CompaRison Over Long Language Sequences
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Measuring Massive Multitask Language Understanding
• Learning to summarize from human feedback
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• CoQA: A Conversational Question Answering Challenge
• Hierarchical Neural Story Generation
• FastText.zip: Compressing text classification models

Related

• Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
• Uncovering Factor Level Preferences to Improve Human-Model Alignment
• Group Preference Optimization: Few-Shot Alignment of Large Language Models