Ministral 3

2026/01/13 by Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian +116 · 3 voices · 7 citations
Computer Science · #cs.CL

paper · pdf

Abstract

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.

Citations

• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
• PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models
• Distillation Scaling Laws
• Scalable-Softmax Is Superior for Attention
• Pixtral 12B
• LLM Pruning and Distillation in Practice: The Minitron Approach
• The Llama 3 Herd of Models
• Compact Language Models via Pruning and Knowledge Distillation
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
• Are We Done with MMLU?
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Direct Language Model Alignment from Online AI Feedback
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• YaRN: Efficient Context Window Extension of Large Language Models
• A Simple and Effective Pruning Approach for Large Language Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• Training language models to follow instructions with human feedback
• Program Synthesis with Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• GLU Variants Improve Transformer
• Root Mean Square Layer Normalization
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• RACE: Large-scale ReAding Comprehension Dataset From Examinations

Cited by

• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• Citation Failure: Definition, Analysis and Efficient Mitigation
• Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation
• SciSchema.org: A Multidisciplinary Collection of Schemas for Structured Scientific Process Descriptions
• Mellum2 Technical Report
• A3 : an Analytical Low-Rank Approximation Framework for Attention
• QTrack: Query-Driven Reasoning for Multi-modal MOT
• The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
• Capabilities Ain't All You Need: Measuring Propensities in AI
• Skill Neologisms: Towards Skill-based Continual Learning

Discussions

• Ministral 3 – pruning via Cascade Distillation [hn, 5 points, 0 comments]
• Ministral 3 was introduced on the 13th of Jan a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three sizes: 3B, 8B, and 14B. [bsky, 4 points, 4 comments]
• From the recent Ministral 3 paper (arxiv.org/pdf/2601.08584) Love these deceptively simple pruning strategies [bsky, 2 points, 0 comments]

Related

• Compact Language Models via Pruning and Knowledge Distillation
• Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation
• Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
• Super Tiny Language Models
• MiniLLM: On-Policy Distillation of Large Language Models