MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks

2025/07/16 by Artem Chervyakov, A. V. Kharitonov, Chervyakov, Artem +43
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Software Engineering (cs.SE) #Software Engineering Research #Teaching and Learning Programming

paper · pdf · doi:10.48550/arxiv.2507.12284

openalex publication_date 2025/07/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most benchmarks prioritize high-level reasoning over executable code and real-world performance, leaving gaps in understanding true capabilities and risks associated with these models in production. To address this issue, we propose MERA Code, a new addition to the MERA benchmark family, specifically focused on evaluating code for the latest code generation LLMs in Russian. This benchmark includes 11 evaluation tasks that span 8 programming languages. Our proposed evaluation methodology features a taxonomy that outlines the practical coding skills necessary for models to complete these tasks. The benchmark comprises an open-source codebase for users to conduct MERA assessments, a scoring system compatible with various programming environments, and a platform featuring a leaderboard and submission system. We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages. We are publicly releasing MERA to guide future research, anticipate groundbreaking features in model development, and standardize evaluation procedures.

Citations

• gpt-oss-120b & gpt-oss-20b Model Card
• GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
• StRuCom: A Novel Dataset of Structured Code Comments in Russian
• YABLoCo: Yet Another Benchmark for Long Context Code Generation
• Correctness Assessment of Code Generated by Large Language Models Using Internal Representations
• Facilitating large language model Russian adaptation with Learned Embedding Propagation
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• A Survey on Large Language Models for Code Generation
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• MERA: A Comprehensive LLM Evaluation in Russian
• TACO: Topics in Algorithmic COde generation dataset
• Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• CodeScore: Evaluating Code Generation by Learning Code Execution
• Multi-lingual Evaluation of Code Generation Models
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
• Making Pre-trained Language Models Better Few-shot Learners
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Unit Test Case Generation with Transformers and Focal Context

Related

• MERA: A Comprehensive LLM Evaluation in Russian
• UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
• CodeArena: A Collective Evaluation Platform for LLM Code Generation
• Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX
• SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks