Understanding Code Semantics: An Evaluation of Transformer Models in Summarization

2023/10/25 by Debanjan Mondal, Abhilasha Lodha, Mondal, Debanjan +5
Computer Science · #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.16314

openalex publication_date 2023/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable names to explore whether models truly understand code semantics or merely rely on textual cues. We have also introduced adversaries like dead code and commented code across three programming languages (Python, Javascript, and Java) to further scrutinize the model's understanding. Ultimately, our research aims to offer valuable insights into the inner workings of transformer-based LMs, enhancing their ability to understand code and contributing to more efficient software development practices and maintenance workflows.

Related

• A Transformer-based Approach for Source Code Summarization
• Large Language Models for Code Summarization
• Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
• Source Code Summarization in the Era of Large Language Models
• Exploring Large Language Models for Code Explanation