Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information

2017/04/24 by Trang Tran, Tran, Trang, Shubham Toshniwal +9
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sound (cs.SD) #cs.CL #cs.LG #cs.SD

paper · pdf · doi:10.48550/arxiv.1704.07287

Accepted in NAACL HLT 2018

arxiv created 2018/04/15 · arxiv updated 2018/04/17

Abstract

In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses. For automatically parsing spoken utterances, we introduce a model that integrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features. We find that different types of acoustic-prosodic features are individually helpful, and together give statistically significant improvements in parse and disfluency detection F1 scores over a strong text-only baseline. For this study with known sentence boundaries, error analyses show that the main benefit of acoustic-prosodic features is in sentences with disfluencies, attachment decisions are most improved, and transcription errors obscure gains from prosody.

Related

• Assessing the Use of Prosody in Constituency Parsing of Imperfect Transcripts
• On the Role of Style in Parsing Speech with Neural Models
• Neural Constituency Parsing of Speech Transcripts
• Improving End-to-End SLU performance with Prosodic Attention and Distillation
• Prosodic segmentation for parsing spoken dialogue