JCAPT: A Joint Modeling Approach for CAPT

2025/06/24 by Tzu-Hsuan Yang, Yang, Tzu-Hsuan, Yueying He +3
Business, Management and Accounting · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Business Process Modeling and Analysis #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.19315

openalex publication_date 2025/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Effective pronunciation feedback is critical in second language (L2) learning, for which computer-assisted pronunciation training (CAPT) systems often encompass two key tasks: automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD). Recent work has shown that joint modeling of these two tasks can yield mutual benefits. Our unified framework leverages Mamba, a selective state space model (SSM), while integrating phonological features and think token strategies to jointly enhance interpretability and fine-grained temporal reasoning in APA and MDD. To our knowledge, this is the first study to combine phonological attribution, SSM-based modeling, and prompting in CAPT. A series of experiments conducted on the speechocean762 benchmark demonstrate that our model consistently outperforms prior methods, particularly on the MDD task.

Citations

• Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss
• MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba
• Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Preserving Phonemic Distinctions for Ordinal Regression: A Novel Loss Function for Automatic Pronunciation Assessment
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment
• Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Phone-level pronunciation scoring and assessment for interactive language learning

Related

• MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling
• Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss
• English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM
• Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training
• Improved Mispronunciation detection system using a hybrid CTC-ATT based approach for L2 English speakers