The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

2025/01/16 by Jonathan Katzy, Katzy, Jonathan, Razvan Mihai Popescu +5 · 3 citations
Computer Science · #Artificial intelligence #Code (set theory) #Computer science #Contamination #Heap (data structure) #Natural Language Processing Techniques #Natural language processing #Programming language #Topic Modeling

paper · pdf · doi:10.1109/forge66646.2025.00025

openalex publication_date 2025/04/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.

Citations

• The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Related

• The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
• The Stack: 3 TB of permissively licensed source code
• FullStack Bench: Evaluating LLMs as Full Stack Coders
• A Survey on Data Contamination for Large Language Models
• A Systematic Evaluation of Large Language Models of Code