Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation

2023/09/12 by Pedro Valero‐Lara, Alexis Huante, Valero-Lara, Pedro +11 · 8 citations
Computer Science · #Advanced Data Storage Technologies #Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Parallel #Parallel Computing and Optimization Techniques #Programming Languages (cs.PL) #Software Engineering (cs.SE) #Topic Modeling #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2309.07103

openalex publication_date 2023/09/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.

Citations

• Julia: A Fresh Approach to Numerical Computing
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Llama 2: Open Foundation and Fine-Tuned Chat Models

Cited by

• LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation
• Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms
• VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMs
• EPIC: Generative AI Platform for Accelerating HPC Operational Data Analytics
• LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests
• CodegenBench: Can LLMs Write Efficient Code Across Architectures?
• Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation
• AI Assistants to Enhance and Exploit the PETSc Knowledge Base

Related

• KernelBench: Can LLMs Write Efficient GPU Kernels?
• LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks
• CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization
• Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels
• CUDA-LLM: LLMs Can Write Efficient CUDA Kernels