ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling

2025/07/10 by Huixian ZHANG, Zhang, Hanlong, Jingsheng Yang +6
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Magnetic confinement fusion research #Speech Recognition and Synthesis #Target Tracking and Data Fusion in Sensor Networks

paper · pdf · doi:10.48550/arxiv.2507.08877

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments.

Citations

• SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
• HAMburger: Accelerating LLM Inference via Token Smashing
• A Generative Caching System for Large Language Models
• SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
• Asynchronous LLM Function Calling
• GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching
• Learning Harmonized Representations for Speculative Sampling
• Efficient Prompt Caching via Embedding Similarity
• Accelerating LLM Inference with Staged Speculative Decoding
• Fast Inference from Transformers via Speculative Decoding
• Language Models are Few-Shot Learners
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Distilling the Knowledge in a Neural Network

Related

• Hammer: Robust Function-Calling for On-Device Language Models via Function Masking
• Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
• Alopex: A Computational Framework for Enabling On-Device Function Calls with LLMs
• Benchmarking Floworks against OpenAI & Anthropic: A Novel Framework for Enhanced LLM Function Calling
• Octopus: On-device language model for function calling of software APIs