An approach to measuring the performance of Automatic Speech Recognition (ASR) models in the context of Large Language Model (LLM) powered applications

2025/07/22 by Sujith Pulikodan, Pulikodan, Sujith, K Sahapthan +7 · 3 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.16456

openalex publication_date 2025/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Automatic Speech Recognition (ASR) plays a crucial role in human-machine interaction and serves as an interface for a wide range of applications. Traditionally, ASR performance has been evaluated using Word Error Rate (WER), a metric that quantifies the number of insertions, deletions, and substitutions in the generated transcriptions. However, with the increasing adoption of large and powerful Large Language Models (LLMs) as the core processing component in various applications, the significance of different types of ASR errors in downstream tasks warrants further exploration. In this work, we analyze the capabilities of LLMs to correct errors introduced by ASRs and propose a new measure to evaluate ASR performance for LLM-powered applications.

Cited by

• StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speech
• WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue
• AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

Related

• Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study
• Multi-stage Large Language Model Correction for Speech Recognition
• ASR Error Correction using Large Language Models
• Leveraging Large Language Models for Exploiting ASR Uncertainty
• On the Robust Approximation of ASR Metrics