SeamlessEdit: Background Noise Aware Zero-Shot Speech Editing with in-Context Enhancement

2025/05/20 by Kuan-Yu Chen, Jeng-Lin Li, Chen, Kuan-Yu +4
Computer Science · #68T45 #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #H.5.5 #I.2.7 #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.14066

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the fast development of zero-shot text-to-speech technologies, it is possible to generate high-quality speech signals that are indistinguishable from the real ones. Speech editing, including speech insertion and replacement, appeals to researchers due to its potential applications. However, existing studies only considered clean speech scenarios. In real-world applications, the existence of environmental noise could significantly degrade the quality of generation. In this study, we propose a noise-resilient speech editing framework, SeamlessEdit, for noisy speech editing. SeamlessEdit adopts a frequency-band-aware noise suppression module and an in-content refinement strategy. It can well address the scenario where the frequency bands of voice and background noise are not separated. The proposed SeamlessEdit framework outperforms state-of-the-art approaches in multiple quantitative and qualitative evaluations.

Citations

• Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions
• E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
• EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
• XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• AttentionStitch: How Attention Solves the Speech Editing Problem
• Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
• uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models
• FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• Diffusion-based Generative Speech Source Separation
• SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion
• WHAM!: Extending Speech Separation to Noisy Environments

Related

• Speech Editing -- a Summary
• An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
• Towards zero-shot Text-based voice editing using acoustic context conditioning, utterance embeddings, and reference encoders
• FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
• SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis