Natural Language-based Assessment of L2 Oral Proficiency using LLMs

2025/07/14 by Bannò, Stefano, Ma, Rao, Qian, Mengjie +3 · 1 citation
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.10200

Abstract

Natural language-based assessment (NLA) is an approach to second language assessment that uses instructions - expressed in the form of can-do descriptors - originally intended for human examiners, aiming to determine whether large language models (LLMs) can interpret and apply them in ways comparable to human assessment. In this work, we explore the use of such descriptors with an open-source LLM, Qwen 2.5 72B, to assess responses from the publicly available S&I Corpus in a zero-shot setting. Our results show that this approach - relying solely on textual information - achieves competitive performance: while it does not outperform state-of-the-art speech LLMs fine-tuned for the task, it surpasses a BERT-based model trained specifically for this purpose. NLA proves particularly effective in mismatched task settings, is generalisable to other data types and languages, and offers greater interpretability, as it is grounded in clearly explainable, widely applicable language descriptors.

Citations

• Qwen2.5 Technical Report
• Speak & Improve Challenge 2025: Tasks and Baseline Systems
• Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback
• Qwen2-Audio Technical Report
• Can GPT-4 do L2 analytic assessment?
• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• Proficiency assessment of L2 spoken English using wav2vec 2.0
• Emergent Abilities of Large Language Models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Opportunities and Challenges of LLMs in Education: An NLP Perspective

Related

• Assessment of L2 Oral Proficiency using Speech Large Language Models
• L2 proficiency assessment using self-supervised speech representations
• Spoken Grammar Assessment Using LLM
• Automatic Proficiency Assessment in L2 English Learners
• TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs