Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer

2023/12/10 by Yongheng Deng, Deng, Yongheng, Ziqing Qiao +7 · 6 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.05842

openalex publication_date 2023/12/10 · openalex created_date 2023/12/13 · openalex updated_date 2026/07/28

Abstract

While large language models (LLMs) are empowered with broad knowledge, their task-specific performance is often suboptimal. It necessitates fine-tuning LLMs with task-specific data, but such data may be inaccessible due to privacy concerns. In this paper, we propose a novel approach to enhance LLMs with smaller language models (SLMs) that are trained on clients using their private task-specific data. To enable mutual enhancement between LLMs and SLMs, we propose CrossLM, where the SLMs promote the LLM to generate task-specific high-quality data, and both the LLM and SLMs are enhanced with the generated data. We evaluate CrossLM using publicly accessible language models across a range of benchmark tasks. The results demonstrate that CrossLM significantly enhances the task-specific performance of SLMs on clients and the LLM on the cloud server simultaneously while preserving the LLM's generalization capability.

Cited by

• FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
• Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
• Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis
• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
• A Survey on Collaborative Mechanisms Between Large and Small Language Models
• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

Related

• A Structure-Agnostic Co-Tuning Framework for LLMs and SLMs in Cloud-Edge Systems
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness
• FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models
• Large-Small Model Collaboration for Enhancing Edge-Deployed Small Models
• Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data