Cross-Lingual Conversational Speech Summarization with Large Language Models

2024/08/12 by Max Nelson, Shannon Wotherspoon, Nelson, Max +7
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2408.06484

openalex publication_date 2024/08/12 · openalex created_date 2024/09/11 · openalex updated_date 2026/07/28

Abstract

Cross-lingual conversational speech summarization is an important problem, but suffers from a dearth of resources. While transcriptions exist for a number of languages, translated conversational speech is rare and datasets containing summaries are non-existent. We build upon the existing Fisher and Callhome Spanish-English Speech Translation corpus by supplementing the translations with summaries. The summaries are generated using GPT-4 from the reference translations and are treated as ground truth. The task is to generate similar summaries in the presence of transcription and translation errors. We build a baseline cascade-based system using open-source speech recognition and machine translation models. We test a range of LLMs for summarization and analyze the impact of transcription and translation errors. Adapting the Mistral-7B model for this task performs significantly better than off-the-shelf models and matches the performance of GPT-4.

Related

• Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation
• SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization
• Zero-Shot Cross-Lingual Summarization via Large Language Models
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization
• The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation