Step-Audio-EditX Technical Report

2025/11/05 by Chao Yan, Yan, Chao, Boyong Wu +23 · 1 citation
Psychology · Computer Science · #Emotion and Mood Recognition #Music and Audio Processing #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2511.03601

Abstract

We present Step-Audio-EditX, the first open-source LLM-based audio model excelling at expressive and iterative audio editing encompassing emotion, speaking style, and paralinguistics alongside robust zero-shot text-to-speech (TTS) capabilities. Our core innovation lies in leveraging only large-margin synthetic data, which circumvents the need for embedding-based priors or auxiliary modules. This large-margin learning approach enables both iterative control and high expressivity across voices, and represents a fundamental pivot from the conventional focus on representation-level disentanglement. Evaluation results demonstrate that Step-Audio-EditX surpasses both MiniMax-2.6-hd and Doubao-Seed-TTS-2.0 in emotion editing and other fine-grained control tasks.

Citations

• FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
• NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization
• Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
• Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
• IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• Autoregressive Speech Synthesis without Vector Quantization
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech
• WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model
• VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training
• Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis

Cited by

• MultiAPI Spoof: A Multi-API Dataset and Local-Attention Network for Speech Anti-spoofing Detection

Related

• Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
• Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model
• SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• Step-Audio 2 Technical Report