Benchmarking Complex Instruction-Following with Multiple Constraints Composition

2024/07/04 by Bosi Wen, Wen, Bosi, Pei Ke +25 · 67 citations
Computer Science · #Benchmarking #Business #Composition (language) #Computer science #Intelligent Tutoring Systems and Adaptive Learning #Linguistics #Marketing #Philosophy

paper · pdf · doi:10.48550/arxiv.2407.03978

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/07/04 · openalex created_date 2024/07/09 · openalex updated_date 2026/07/28

Abstract

Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition.

Cited by

• Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
• Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs
• WIRE: Profiling Witnessed Within-Policy Instruction Collisions in LLM Agents
• CIFE: Code Instruction-Following Evaluation
• Revisiting the Reliability of Language Models in Instruction-Following
• Financial Instruction Following Evaluation (FIFE)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues
• ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
• AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
• LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
• When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs
• IF-VidCap: Can Video Caption Models Follow Instructions?
• Boosting Instruction Following at Scale
• Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
• Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systems
• Analyzing and Internalizing Complex Policy Documents for LLM Agents
• Format Inertia: A Failure Mechanism of LLMs in Medical Pre-Consultation
• SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures
• Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models
• Mem-α: Learning Memory Construction via Reinforcement Learning
• RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
• When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following
• TRUEBench: Can LLM Response Meet Real-world Constraints as Productivity Assistant?
• IFHierBench: Hierarchical Instruction Following for Large Language Models
• Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?
• AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions
• RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
• Routine: A Structural Planning Framework for LLM Agent System in Enterprise
• EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models
• How Many Instructions Can LLMs Follow at Once?
• Fine-tuning on simulated data outperforms prompting for agent tone of voice
• Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
• A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
• SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents
• What Makes a Good Natural Language Prompt?
• AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
• AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
• Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
• FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes
• EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writing
• HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization
• NavBench: Probing Multimodal Large Language Models for Embodied Navigation
• Guiding Generative Storytelling with Knowledge Graphs
• Reverse Preference Optimization for Complex Instruction Following
• RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data
• Training with Pseudo-Code for Instruction Following
• AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios
• Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
• Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering
• Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
• GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents
• When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs
• Answer Presence Drives RAG Rewriting Gains
• A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models
• LookAlike: Consistent Distractor Generation in Math MCQs
• From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves
• Power and Limitations of Aggregation in Compound AI Systems
• Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capability
• Buy versus Build an LLM: A Decision Framework for Governments
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios

Related

• From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models
• A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models
• FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
• IFHierBench: Hierarchical Instruction Following for Large Language Models
• InFoBench: Evaluating Instruction Following Ability in Large Language Models