2021/11/16 by Takatomo Kano, Atsunori Ogawa, Kano, Takatomo +5 · 1 citation
Computer Science · Engineering · #Artificial intelligence #Audio and Speech Processing (eess.AS) #Automatic summarization #Cascade #Computation and Language (cs.CL) #Computer science #Embedding #Encoder #Exploit #FOS: Computer and information sciences #FOS: Electrical engineering #Language model #Natural Language Processing Techniques #Natural language processing #Speech Recognition and Synthesis #Speech recognition #Topic Modeling #Transformer #cs.CL #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2111.08201
published in arXiv (Cornell University) (Cornell University)
arxiv created 2021/11/16 · openalex publication_date 2021/11/16 · arxiv updated 2021/11/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper. These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.