A Simple, Yet Effective Approach to Finding Biases in Code Generation

2022/10/31 by Spyridon Mouselinos, Mateusz Malinowski, Mouselinos, Spyridon +3 · 1 citation
Computer Science · #Advanced Malware Detection Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Programming Languages (cs.PL) #Software Engineering Research #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2211.00609

openalex publication_date 2022/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, high-performing code generation systems based on large language models have surfaced. They are trained on massive corpora containing much more natural text than actual executable computer code. This work shows that current code generation systems exhibit undesired biases inherited from their large language model backbones, which can reduce the quality of the generated code under specific circumstances. To investigate the effect, we propose the "block of influence" concept, which enables a modular decomposition and analysis of the coding challenges. We introduce an automated intervention mechanism reminiscent of adversarial testing that exposes undesired biases through the failure modes of the models under test. Finally, we demonstrate how our framework can be used as a data transformation technique during fine-tuning, acting as a mitigation strategy for these biases.

Cited by

• MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks

Related

• On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code
• Capturing Failures of Large Language Models via Human Cognitive Biases
• Evaluating How Fine-tuning on Bimodal Data Effects Code Generation
• AKD : Adversarial Knowledge Distillation For Large Language Models Alignment on Coding tasks
• FairCoder: Evaluating Social Bias of LLMs in Code Generation