MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training

2025/02/17 by Hui Huang, Jiaheng Liu, Huang, Hui +13 · 4 citations
Psychology · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Education and Critical Thinking Development #FOS: Computer and information sciences #Innovative Teaching and Learning Methods #Problem and Project Based Learning

paper · pdf · doi:10.48550/arxiv.2502.11541

openalex publication_date 2025/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods.

Cited by

• STAIF: A Stage-wise Optimization for Complex Instruction Following
• Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following

Related

• Automatic Pair Construction for Contrastive Post-training
• Align²LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
• IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization
• Improving the Robustness of Large Language Models via Consistency Alignment
• Sequence-level Large Language Model Training with Contrastive Preference Optimization