Bielik v3 Small: Technical Report

2025/05/05 by Krzysztof Ociepa, Łukasz Flis, Ociepa, Krzysztof +7
Computer Science · #68T50 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.02550

openalex publication_date 2025/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Bielik v3, a series of parameter-efficient generative text models (1.5B and 4.5B) optimized for Polish language processing. These models demonstrate that smaller, well-optimized architectures can achieve performance comparable to much larger counterparts while requiring substantially fewer computational resources. Our approach incorporates several key innovations: a custom Polish tokenizer (APT4) that significantly improves token efficiency, Weighted Instruction Cross-Entropy Loss to balance learning across instruction types, and Adaptive Learning Rate that dynamically adjusts based on training progress. Trained on a meticulously curated corpus of 292 billion tokens spanning 303 million documents, these models excel across multiple benchmarks, including the Open PL LLM Leaderboard, Complex Polish Text Understanding Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter model achieves results competitive with models 2-3 times its size, while the 1.5B model delivers strong performance despite its extremely compact profile. These advances establish new benchmarks for parameter-efficient language modeling in less-represented languages, making high-quality Polish language AI more accessible for resource-constrained applications.

Citations

• Evaluating Polish linguistic and cultural competency in large language models
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
• Instruction Tuning With Loss Over Instructions
• GPT-4 passes most of the 297 written Polish Board Certification Examinations
• Simple and Scalable Strategies to Continually Pre-train Large Language Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
• EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
• OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining
• StyloMetrix: An Open-Source Multilingual Tool for Representing Stylometric Vectors
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
• Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• PolQA: Polish Question Answering Dataset
• Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases
• Continual Learning with Foundation Models: An Empirical Study of Latent Replay
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• KLEJ: Comprehensive Benchmark for Polish Language Understanding
• GLU Variants Improve Transformer
• Evaluation of Sentence Representations in Polish
• From English To Foreign Languages: Transferring Pre-trained Language Models
• Attention Is All You Need
• Language Modeling with Gated Convolutional Networks

Related

• Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation
• Bielik 11B v2 Technical Report
• Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish
• PLLuM: A Family of Polish Large Language Models
• Pre-training Polish Transformer-based Language Models at Scale