The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks

2024/02/27 by Ashwin Prasad Shivarpatna Venkatesh, Samkutty Sabu, Venkatesh, Ashwin Prasad Shivarpatna +7 · 1 citation
Computer Science · #FOS: Computer and information sciences #Software Engineering (cs.SE) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.17679

openalex publication_date 2024/02/27 · openalex created_date 2024/03/05 · openalex updated_date 2026/07/28

Abstract

The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.

Cited by

• Advancing Automated In-Isolation Validation in Repository-Level Code Translation

Related

• An Empirical Study of Large Language Models for Type and Call Graph Analysis in Python and JavaScript
• Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness
• TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
• Do Code LLMs Do Static Analysis?
• The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Models