Trillion 7B Technical Report

2025/04/21 by Sungjun Han, Han, Sungjun, Juyoung Suk +13 · 4 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.15431

openalex publication_date 2025/04/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours ($148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency.

Citations

• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training
• How does a Language-Specific Tokenizer affect LLMs?
• Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap
• The broader spectrum of in-context learning
• VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models
• Code-Switching Curriculum Learning for Multilingual Transfer in LLMs
• How to Train Long-Context Language Models (Effectively)
• How Does Code Pretraining Affect Language Model Task Performance?
• EXAONE 3.0 7.8B Instruction Tuned Language Model
• Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
• Does your data spark joy? Performance gains from domain upsampling at the end of training
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• Better & Faster Large Language Models via Multi-token Prediction
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Understanding Emergent Abilities of Language Models from the Loss Perspective
• How do Large Language Models Handle Multilingualism?
• Analysing The Impact of Sequence Composition on Language Model Pre-Training
• KMMLU: Measuring Massive Multitask Language Understanding in Korean
• Do Llamas Work in English? On the Latent Language of Multilingual Transformers
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Instruction-Following Evaluation for Large Language Models
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Effective Long-Context Scaling of Foundation Models
• HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• KOBEST: Korean Balanced Evaluation of Significant Tasks
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Training Verifiers to Solve Math Word Problems
• The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• mT5: A massively multilingual pre-trained text-to-text transformer
• Language Models are Few-Shot Learners
• GLU Variants Improve Transformer
• Scaling Laws for Neural Language Models
• Root Mean Square Layer Normalization
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need

Cited by

• KORMo: Korean Open Reasoning Model for Everyone
• Predicting LLM Reasoning Performance with Small Proxy Model
• Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks

Related

• Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models
• RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining
• KMMLU: Measuring Massive Multitask Language Understanding in Korean
• Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
• Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources