SegTune: Structured and Fine-Grained Control for Song Generation

2025/10/21 by Pengfei Cai, Cai, Pengfei, Joanna Wang +15
Computer Science · #Artificial Intelligence in Games #Coherence (philosophical gambling strategy) #Construct (python library) #Control (management) #Controllability #FOS: Computer and information sciences #Limiting #Lyrics #Music Technology and Sound Studies #Music and Audio Processing #Musical #Pipeline (software) #Sound (cs.SD)

paper · pdf · doi:10.48550/arxiv.2510.18416

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/08/05

Abstract

Recent advancements in song generation have shown promising results in generating songs from lyrics and/or global text prompts. However, most existing systems lack the ability to model the temporally varying attributes of songs, limiting fine-grained control over musical structure and dynamics. In this paper, we propose SegTune, a non-autoregressive framework for structured and controllable song generation. SegTune enables segment-level control by allowing users or large language models to specify local musical descriptions aligned to song sections.The segmental prompts are injected into the model by temporally broadcasting them to corresponding time windows, while global prompts influence the whole song to ensure stylistic coherence. To obtain accurate segment durations and enable precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamped lyrics in LRC format. We further construct a large-scale data pipeline for collecting high-quality songs with aligned lyrics and prompts, and propose new evaluation metrics to assess segment-level alignment and vocal attribute consistency. Experimental results show that SegTune achieves superior controllability and musical coherence compared to existing baselines. See https://cai525.github.io/SegTune_demo for demos of our work.

Citations

• Qwen3-Omni Technical Report
• JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment
• DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization
• FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generation
• Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
• LeVo: High-Quality Song Generation with Multi-Preference Alignment
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• ACE-Step: A Step Towards Music Generation Foundation Model
• SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
• Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation
• YuE: Scaling Open Foundation Models for Long-Form Music Generation
• DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
• Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
• FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration
• MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization
• SongCreator: Lyrics-based Universal Song Generation
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• Understanding the Impact of Negative Prompts: When and How Do They Take Effect?
• Long-form music generation with latent diffusion
• Music ControlNet: Multiple Time-varying Controls for Music Generation
• MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
• All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio
• Simple and Controllable Music Generation
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Efficient Neural Music Generation
• LLaMA: Open and Efficient Foundation Language Models
• Improving and generalizing flow-based generative models with minibatch optimal transport
• Scalable Diffusion Models with Transformers
• Robust Speech Recognition via Large-Scale Weak Supervision
• Hybrid Transformers for Music Source Separation
• Flow Matching for Generative Modeling
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Denoising Diffusion Probabilistic Models
• Jukebox: A Generative Model for Music

Related

• JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment
• Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering
• Unsupervised Melody-to-Lyric Generation
• SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
• CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls