How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
2025/09/19 by Kangtao Lv, Haibin Chen, Lv, Kangtao +13 · 10 voices
#cs.CL #cs.AI
paper · pdf · doi:10.48550/arxiv.2509.19371
Abstract
Large language models (LLMs) have attracted significant attention due to their impressive general capabilities across diverse downstream tasks. However, without domain-specific optimization, they often underperform on specialized knowledge benchmarks and even produce hallucination. Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance. A critical challenge lies in balancing this infusion trade-off: injecting too little domain-specific data yields insufficient specialization, whereas excessive infusion triggers catastrophic forgetting of previously acquired knowledge. In this work, we focus on the phenomenon of memory collapse induced by over-infusion. Through systematic experiments, we make two key observations, i.e. 1) Critical collapse point: each model exhibits a threshold beyond which its knowledge retention capabilities sharply degrade. 2) Scale correlation: these collapse points scale consistently with the model's size. Building on these insights, we propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. Extensive experiments across different model sizes and pertaining token budgets validate both the effectiveness and generalizability of our scaling law.
Citations
Discussions
- Knowledge Infusion Scaling Law for Pre-Training Large Language Models https://arxiv.org/abs/2509.19371 (https://news.ycombinator.com/item?id=45474900) [bsky, 0 points, 0 comments]
- How to inject knowledge efficiently? Knowledge infusion scaling law for LLMs https://arxiv.org/abs/2509.19371 (https://news.ycombinator.com/item?id=45474900) [bsky, 0 points, 0 comments]
- https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3m2jbnch6yjy2 [bsky, 0 points, 0 comments]
- How to inject knowledge efficiently? Knowledge infusion scaling law for LLMs [bsky, 0 points, 0 comments]
- Knowledge Infusion Scaling Law for Pre-Training Large Language Models #HackerNews https://arxiv.org/abs/2509.19371 [bsky, 0 points, 0 comments]
- Knowledge Infusion Scaling Law for Pre-Training Large Language Models https://arxiv.org/abs/2509.19371 [bsky, 0 points, 0 comments]
- ⚡ Hackernews Top story: Knowledge Infusion Scaling Law for Pre-Training Large Language Models [bsky, 0 points, 0 comments]
- 研究者が、事前学習中に外部知識を取り込むことで大規模言語モデルの性能が向上することを示しました。 知識注入量とモデルサイズの関係を規則化し、拡張性を数式で表したスケーリング法則が発見されました。 この法則は、モデルが大きくなるほど知識効果が倍増することを示し、実験で確認されています。 arxiv.org #news [bsky, 0 points, 0 comments]
- How to inject knowledge efficiently? Knowledge infusion scaling law for LLMs arxiv.org/abs/2509.19371 Discuss: www.sqox.com/c/?id=c10bf0... [bsky, 0 points, 0 comments]
- https://arxiv.org/abs/2509.19371 大規模言語モデルの事前学習に関する研究論文です。 知識注入のスケーリング則について考察しています。 効率的な知識注入方法を提案している可能性があります。 [bsky, 0 points, 0 comments]
Related