Towards a Small Language Model Lifecycle Framework
2025/06/09 by Parsa Miraghaei, Sergio Moreschini, Miraghaei, Parsa +5
Computer Science · Medicine · Social Sciences · #Artificial Intelligence in Healthcare and Education #Computational and Text Analysis Methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2506.07695
openalex publication_date 2025/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Background: The growing demand for efficient and deployable language models has led to increased interest in Small Language Models (SLMs). However, existing research remains fragmented, lacking a unified lifecycle perspective. Objective: This study aims to define a comprehensive lifecycle framework for SLMs by synthesizing insights from academic literature and practitioner sources. Method: We conducted a comprehensive survey of 36 works, analyzing and categorizing lifecycle-relevant techniques. Results: We propose a modular lifecycle model structured into main, optional, and cross-cutting components. The model captures key interconnections across stages, supporting method reuse, co-adaptation, and lifecycle-awareness. Conclusion: Our framework provides a coherent foundation for developing and maintaining SLMs, bridging theory and practice, and guiding future research and tool development.
Citations
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
- Qwen2.5 Technical Report
- A Survey on LLM Inference-Time Self-Improvement
- Phi-4 Technical Report
- A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
- A Survey of Small Language Models
- Computational Bottlenecks of Training Small-scale Large Language Models
- Small Language Models: Survey, Measurements, and Insights
- What is the Role of Small Models in the LLM Era: A Survey
- On-Device Language Models: A Comprehensive Review
- The Llama 3 Herd of Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Federated Distillation: A Survey
- Arcee's MergeKit: A Toolkit for Merging Large Language Models
- Datasets for Large Language Models: A Comprehensive Survey
- LLM Inference Unveiled: Survey and Roofline Model Insights
- A Survey on Data Selection for Language Models
- A Survey on Knowledge Distillation of Large Language Models
- Model Compression and Efficient Inference for Large Language Models: A Survey
- Large Language Models: A Survey
- A Survey on Transformer Compression
- Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models
- Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
- Efficient Large Language Models: A Survey
- A Survey on Model Compression for Large Language Models
- A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
- A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency
- LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
- Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Review
- A Survey of Large Language Models
- Machine Learning Operations (MLOps): Overview, Definition, and Architecture
- Training Compute-Optimal Large Language Models
- Efficient Transformers: A Survey
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
- HAQ: Hardware-Aware Automated Quantization with Mixed Precision
- Guidelines for including grey literature and conducting multivocal\n literature reviews in software engineering
- Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)
Related