Patwary, Mostofa
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2019/09/17 by Mohammad Shoeybi, Mostofa Patwary, Shoeybi, Mohammad +9 · 319 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis
- Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
2021/04/09 by Deepak Narayanan, Narayanan, Deepak, Mohammad Shoeybi +21 · 108 citations
Computer Science · #Computation and Language (cs.CL) #Distributed #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Parallel #Topic Modeling #and Cluster Computing (cs.DC)
- StarCoder 2 and The Stack v2: The Next Generation
2024/02/29 by Lozhkov, Anton, Li, Raymond, Allal, Loubna Ben +63 · 95 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Software Engineering (cs.SE)
- Llama-Nemotron: Efficient Reasoning Models
2025/05/02 by Akhiad Bercovich, Itay Levy, Bercovich, Akhiad +224 · 1 voice · 29 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2022/01/28 by Shaden Smith, Smith, Shaden, Mostofa Patwary +36 · 24 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
- Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
2024/12/03 by Dan Su, Kezhi Kong, Su, Dan +15 · 49 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Image Processing and 3D Reconstruction #Natural Language Processing Techniques #Semantic Web and Ontologies
- Factuality Enhanced Language Models for Open-Ended Text Generation
2022/06/09 by Nayeon Lee, Lee, Nayeon, Peng Xu +9 · 20 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling
- Compact Language Models via Pruning and Knowledge Distillation
2024/07/19 by Saurav Muralidharan, Muralidharan, Saurav, Sharath Turuvekere Sreenivas +15 · 27 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques
- Nemotron-4 340B Technical Report
2024/06/17 by Nvidia, :, Adler, Bo +80 · 20 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- LLM Pruning and Distillation in Practice: The Minitron Approach
2024/08/21 by Sharath Turuvekere Sreenivas, Sreenivas, Sharath Turuvekere, Saurav Muralidharan +23 · 18 citations
Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Extraction and Separation Processes #FOS: Computer and information sciences #Machine Learning (cs.LG) #Metallurgical Processes and Thermodynamics #Mineral Processing and Grinding
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
2025/04/04 by NVIDIA, :, Blakeman, Aaron +198 · 26 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- Nemotron-4 15B Technical Report
2024/02/26 by Jupinder Parmar, Shrimai Prabhumoye, Parmar, Jupinder +51 · 1 voice · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CL #cs.LG
- BioMegatron: Larger Biomedical Domain Language Model
2020/10/12 by Hoo-Chang Shin, Yang Zhang, Shin, Hoo-Chang +11 · 5 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
- Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
2025/04/17 by Diao, Shizhe, Yang, Yu, Fu, Yonggan +12 · 15 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences
- Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
2024/07/09 by Jupinder Parmar, Sanjev Satheesh, Parmar, Jupinder +7 · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques
- Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
2024/12/18 by Shrimai Prabhumoye, Feng, Steven, Prabhumoye, Shrimai +10 · 9 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications
- Training Question Answering Models From Synthetic Data
2020/02/22 by Puri, Raul, Spring, Ryan, Patwary, Mostofa +2 · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences
- NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
2025/08/20 by NVIDIA, Aarti Basant, : +305 · 23 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques
- MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
2020/10/02 by Xu, Peng, Patwary, Mostofa, Shoeybi, Mohammad +4 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences
- Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
2025/04/15 by Akter, Syeda Nahida, Prabhumoye, Shrimai, Novikov, Matvei +8 · 9 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
2022/02/08 by Boxin Wang, Wang, Boxin, Ping, Wei +14 · 2 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Adversarial Robustness in Machine Learning
- Pretraining Large Language Models with NVFP4
2025/09/29 by NVIDIA, Abecassis, Felix, Agrusa, Anjulie +86 · 11 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- Coloring Big Graphs with AlphaGoZero
2019/02/26 by Jiayi Huang, Md. Mostofa Ali Patwary, Huang, Jiayi +3 · 2 citations
Computer Science · #Graph Theory and Algorithms #Graph Labeling and Dimension Problems #Advanced Graph Theory Research
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
2024/10/15 by Syeda Nahida Akter, Shrimai Prabhumoye, Akter, Syeda Nahida +13 · 4 citations
Computer Science · #Natural Language Processing Techniques #Mathematics, Computing, and Information Processing #Semantic Web and Ontologies
- Multi-Agent Evolve: LLM Self-Improve through Co-evolution
2025/10/27 by Chen, Yixing, Wang, Yiding, Zhu, Siqi +5 · 8 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences
- Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning
2025/04/06 by Ximing Lu, Seungju Han, Lu, Ximing +19 · 5 citations
Computer Science · #Intelligent Tutoring Systems and Adaptive Learning #Explainable Artificial Intelligence (XAI) #Advanced Graph Neural Networks
- NVIDIA Nemotron 3: Efficient and Open Intelligence
2025/12/24 by NVIDIA, :, Blakeman, Aaron +356 · 4 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems
2019/09/25 by Adam Rupe, Rupe, Adam, Nalini Kumar +18 · 1 voice
Computer Science · Physics and Astronomy · #Computational Physics (physics.comp-ph) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Performance (cs.PF) #cs.LG #cs.PF #physics.comp-ph
- Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
2025/04/15 by Ali Taghibakhshi, Taghibakhshi, Ali, Sharath Turuvekere Sreenivas +33 · 5 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis
- Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
2025/12/23 by NVIDIA, :, Aaron Blakeman +623 · 1 voice · 2 citations
#cs.CL #cs.AI #cs.LG
- Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
2025/09/26 by Syeda Nahida Akter, Shrimai Prabhumoye, Akter, Syeda Nahida +11 · 6 citations
Computer Science · #Intelligent Tutoring Systems and Adaptive Learning
- Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
2025/08/20 by Rabeeh Karimi Mahabadi, Mahabadi, Rabeeh Karimi, Sanjeev Satheesh +9 · 1 voice · 5 citations
#cs.CL #cs.AI #cs.LG
- Data, Data Everywhere: A Guide for Pretraining Dataset Construction
2024/07/08 by Parmar, Jupinder, Prabhumoye, Shrimai, Jennings, Joseph +6 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences
- RLP: Reinforcement as a Pretraining Objective
2025/09/26 by Ali Hatamizadeh, Syeda Nahida Akter, Hatamizadeh, Ali +13 · 4 citations
Psychology · #Behavioral and Psychological Studies