On Adversarial Robustness of Synthetic Code Generation

2021/06/22 by Anand, Mrinal, Kayal, Pratik, Singh, Mayank · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2106.11629

Abstract

Automatic code synthesis from natural language descriptions is a challenging task. We witness massive progress in developing code generation systems for domain-specific languages (DSLs) employing sequence-to-sequence deep learning techniques in the recent past. In this paper, we specifically experiment with AlgoLisp DSL-based generative models and showcase the existence of significant dataset bias through different classes of adversarial examples. We also experiment with two variants of Transformer-based models that outperform all existing AlgoLisp DSL-based code generation baselines. Consistent with the current state-of-the-art systems, our proposed models, too, achieve poor performance under adversarial settings. Therefore, we propose several dataset augmentation techniques to reduce bias and showcase their efficacy using robust experimentation.

Cited by

• Adversarial Attack Classification and Robustness Testing for Large Language Models for Code

Related

• Synthetic Datasets for Neural Program Synthesis
• A Simple, Yet Effective Approach to Finding Biases in Code Generation
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• Code Difference Guided Adversarial Example Generation for Deep Code Models
• On the Reliability and Explainability of Language Models for Program Generation