Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments

2025/10/14 by Ziming Dai, Tuo Zhang, Dai, Ziming +11
Computer Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.15992

openalex publication_date 2025/10/14 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28

Abstract

The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment.

Citations

• ACME: Adaptive Customization of Large Models via Distributed Systems
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• GPT-FL: Generative Pre-trained Model-Assisted Federated Learning
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
• Language Models are Few-Shot Learners

Related

• DDK: Distilling Domain Knowledge for Efficient Large Language Models
• KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
• DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models
• Can a student Large Language Model perform as well as it's teacher?
• GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment