STASE: A spatialized text-to-audio synthesis engine for music generation

2025/09/14 by Chi, Tutti, Letian Gao, Yixiao Zhang +2
Computer Science · #Music and Audio Processing #Music Technology and Sound Studies #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2509.11124

Abstract

While many text-to-audio systems produce monophonic or fixed-stereo outputs, generating audio with user-defined spatial properties remains a challenge. Existing deep learning-based spatialization methods often rely on latent-space manipulations, which can limit direct control over psychoacoustic parameters critical to spatial perception. To address this, we introduce STASE, a system that leverages a Large Language Model (LLM) as an agent to interpret spatial cues from text. A key feature of STASE is the decoupling of semantic interpretation from a separate, physics-based spatial rendering engine, which facilitates interpretable and user-controllable spatial reasoning. The LLM processes prompts through two main pathways: (i) Description Prompts, for direct mapping of explicit spatial information (e.g., "place the lead guitar at 45° azimuth, 10 m distance"), and (ii) Abstract Prompts, where a Retrieval-Augmented Generation (RAG) module retrieves relevant spatial templates to inform the rendering. This paper details the STASE workflow, discusses implementation considerations, and highlights current challenges in evaluating generative spatial audio.

Citations

• ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model
• Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
• MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation
• Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning
• Fast Timing-Conditioned Latent Audio Diffusion
• JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation
• Simple and Controllable Music Generation
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• A Review of Intelligent Music Generation Systems
• BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with\n Depth and Cross Modal Attention
• dEchorate: a Calibrated Room Impulse Response Database for Echo-aware Signal Processing
• Learning Transferable Visual Models From Natural Language Supervision
• Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

Related

• SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing
• BAT: Learning to Reason about Spatial Sounds with Large Language Models
• Can Large Language Models Understand Spatial Audio?
• SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
• Learning Spatially-Aware Language and Audio Embeddings