CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

2024/08/29 by Laurin Wagner, Bernhard Thallinger, Wagner, Laurin +3 · 1 voice · 8 citations
Computer Science · #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2408.16589

openalex publication_date 2024/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open https://github.com/nyrahealth/CrisperWhisper.

Cited by

• Linguistic and Audio Embedding-Based Machine Learning for Alzheimer's Dementia and Mild Cognitive Impairment Detection: Insights from the PROCESS Challenge
• Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
• Talking to Robots: A Practical Examination of Speech Foundation Models for HRI Applications
• Cross-Attention is Half Explanation in Speech-to-Text Models
• Whisper Has an Internal Word Aligner
• Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
• Revealing the Role of Audio Channels in ASR Performance Degradation

Discussions

• CrisperWhisper. "Since our model transcribes verbatim, we also detect and segment other disfluency types, such as repetitions, false starts or partial words." Though intended for use by speech patholo [bsky, 10 points, 2 comments]

Related

• WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
• CarelessWhisper: Turning Whisper into a Causal Streaming Model
• Whisper Has an Internal Word Aligner
• Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing
• Smooth Operators: LLMs Translating Imperfect Hints into Disfluency-Rich Transcripts