HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages

2024/12/19 by A. Chaturvedi, Chaturvedi, Aman, Daniel Nichols +5 · 9 citations
Computer Science · Decision Sciences · #Advanced Data Storage Technologies #Algorithms and Data Compression #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Scientific Computing and Data Management #Software Engineering (cs.SE) #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2412.15178

openalex publication_date 2024/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. Creating specialized models and tools for these domains is crucial towards gaining the benefits of LLMs in areas such as HPC. While previous work has explored HPC-specific models, LLMs still struggle to generate parallel code and it is not at all clear what hurdles are still holding back these LLMs and what must be done to overcome them. In this work, we conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges. Based on our findings we fine-tune and evaluate a specialized HPC LLM that is shown to be the best performing open-source code LLM for parallel code generation to date.

Cited by

• LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation
• OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms
• From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph
• Integrating Performance Tools in Model Reasoning for GPU Kernel Optimization
• UniPar: A Unified LLM-Based Framework for Parallel and Accelerated Code Translation in HPC
• Systematic Evaluation of Optimization Techniques for Long-Context Language Models
• Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codes
• CommBench: Can LLMs Write Correct and Efficient GPU Communication Code?
• RepoOMP: Repository-Aware Hotspot OpenMP Parallelization via Dependency-Aware Context Reduction

Related

• Scope is all you need: Transforming LLMs for HPC Code
• CodegenBench: Can LLMs Write Efficient Code Across Architectures?
• The Landscape and Challenges of HPC Research and LLMs
• MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
• LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation