Understanding LLMs: A Comprehensive Overview from Training to Inference

2024/01/04 by Yiheng Liu, Hao He, Liu, Yiheng +39 · 4 voices · 4 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL

paper · pdf · doi:10.48550/arxiv.2401.02038

arxiv published 2024/01/04 · arxiv updated 2024/01/06

Abstract

The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There's an increasing focus on cost-efficient training and deployment within this context. Low-cost training and deployment of LLMs represent the future development trend. This paper reviews the evolution of large language model training techniques and inference deployment technologies aligned with this emerging trend. The discussion on training includes various aspects, including data preprocessing, training architecture, pre-training tasks, parallel training, and relevant content related to model fine-tuning. On the inference side, the paper covers topics such as model compression, parallel computation, memory scheduling, and structural optimization. It also explores LLMs' utilization and provides insights into their future development.

Cited by

• Predicate Renaming via Large Language Models
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
• A Survey: Towards Privacy and Security in Mobile Large Language Models
• Pruning Large Language Models by Identifying and Preserving Functional Networks

Discussions

• Understanding LLMs: A Comprehensive Overview from Training to Inference [hn, 3 points, 0 comments]
• Understanding LLMs: A Comprehensive Overview from Training to Inference [hn, 2 points, 0 comments]
• Understanding LLMs: A Comprehensive Overview from Training to Inference Yiheng Liu, etc. Source: arxiv.org/abs/2401.02038 [bsky, 1 points, 0 comments]
• The link to the paper. arxiv.org/pdf/2401.02038 [bsky, 0 points, 0 comments]

Related

• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
• LLM Inference Serving: Survey of Recent Advances and Opportunities
• LLM Inference Unveiled: Survey and Roofline Model Insights
• Large Language Models: A Survey
• Training Data for Large Language Model