NotSoTiny: A Large, Living Benchmark for RTL Code Generation

2025/12/23 by Razine Moundir Ghorab, Emanuele Parisi, Ghorab, Razine Moundir +11
Computer Science · #Embedded Systems Design Techniques #Formal Methods in Verification #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2512.20823

Abstract

LLMs have shown early promise in generating RTL code, yet evaluating their capabilities in realistic setups remains a challenge. So far, RTL benchmarks have been limited in scale, skewed toward trivial designs, offering minimal verification rigor, and remaining vulnerable to data contamination. To overcome these limitations and to push the field forward, this paper introduces NotSoTiny, a benchmark that assesses LLM on the generation of structurally rich and context-aware RTL. Built from hundreds of actual hardware designs produced by the Tiny Tapeout community, our automated pipeline removes duplicates, verifies correctness and periodically incorporates new designs to mitigate contamination, matching Tiny Tapeout release schedule. Evaluation results show that NotSoTiny tasks are more challenging than prior benchmarks, emphasizing its effectiveness in overcoming current limitations of LLMs applied to hardware design, and in guiding the improvement of such promising technology.

Citations

• gpt-oss-120b & gpt-oss-20b Model Card
• TuRTLe: A Unified Evaluation of LLMs for RTL Generation
• VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
• A Survey of Research in Large Language Models for Electronic Design Automation
• HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineers
• Qwen2.5 Technical Report
• Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges
• Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation
• The Llama 3 Herd of Models
• AI Benchmark Half-Life in Recursive Corpora: A Theory of Validity Decay under Semantic Leakage and Regeneration
• Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
• VerilogEval: Evaluating Large Language Models for Verilog Code Generation
• Evaluating Large Language Models Trained on Code

Related

• RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solution
• RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects
• TuRTLe: A Unified Evaluation of LLMs for RTL Generation
• ArchXBench: A Complex Digital Systems Benchmark Suite for LLM Driven RTL Synthesis
• Assessing Large Language Models in Generating RTL Design Specifications