APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets

2024/06/26 by Zuxin Liu, Thai Hoang, Liu, Zuxin +31 · 52 citations
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Scientific Computing and Data Management #Software Engineering (cs.SE) #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2406.18518

openalex publication_date 2024/06/26 · openalex created_date 2024/06/28 · openalex updated_date 2026/07/28

Abstract

The advancement of function-calling agent models requires diverse, reliable, and high-quality datasets. This paper presents APIGen, an automated data generation pipeline designed to synthesize verifiable high-quality datasets for function-calling applications. We leverage APIGen and collect 3,673 executable APIs across 21 different categories to generate diverse function-calling datasets in a scalable and structured manner. Each data in our dataset is verified through three hierarchical stages: format checking, actual function executions, and semantic verification, ensuring its reliability and correctness. We demonstrate that models trained with our curated datasets, even with only 7B parameters, can achieve state-of-the-art performance on the Berkeley Function-Calling Benchmark, outperforming multiple GPT-4 models. Moreover, our 1B model achieves exceptional performance, surpassing GPT-3.5-Turbo and Claude-3 Haiku. We release a dataset containing 60,000 high-quality entries, aiming to advance the field of function-calling agent domains. The dataset is available on Huggingface: https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k and the project homepage: https://apigen-pipeline.github.io/

Cited by

• Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents
• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles
• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
• ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs
• Verification-Guided Context Optimization for Tool Calling via Hierarchical LLMs-as-Editors
• GTM: Simulating the World of Tools for AI Agents
• CoRT: Code-integrated Reasoning within Thinking
• PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
• GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards
• Simulating Environments with Reasoning Models for Agent Training
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
• ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset
• NVIDIA Nemotron Nano V2 VL
• SynQuE: Estimating Synthetic Dataset Quality Without Annotations
• Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch
• Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligence
• PORTool: Tool-Use LLM Training with Rewarded Tree
• Automating Benchmark Design
• FunReason-MT Technical Report: Advanced Data Synthesis Solution for Real-world Multi-Turn Tool-use
• MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools
• Language Ranker: A Lightweight Ranking framework for LLM Decoding
• Teaching Language Models to Reason with Tools
• Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
• TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
• How Good Are LLMs at Processing Tool Outputs?
• MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoning
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning
• TokMem: Tokenized Procedural Memory for Large Language Models
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
• Towards General Agentic Intelligence via Environment Scaling
• ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
• PIE: Performance Interval Estimation for Free-Form Generation Tasks
• VehicleWorld: A Highly Integrated Multi-Device Environment for Intelligent Vehicle Interaction
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on τ-bench
• GTool: Graph Enhanced Tool Planning with Large Language Model
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
• UserBench: An Interactive Gym Environment for User-Centric Agents
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• Benchmarking LLM Tool-Use in the Wild
• Routine: A Structural Planning Framework for LLM Agent System in Enterprise
• Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation
• Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky
• MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models
• VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs
• JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent
• Arch-Router: Aligning LLM Routing with Human Preferences
• Towards Pervasive Distributed Agentic Generative AI -- A State of The Art
• Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling

Related

• ToolACE: Winning the Points of LLM Function Calling
• Hammer: Robust Function-Calling for On-Device Language Models via Function Masking
• API Pack: A Massive Multi-Programming Language Dataset for API Call Generation
• Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents