The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling

2025/09/19 by Patrick O’Reilly, O'Reilly, Patrick, Barnett, Julia +10
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Human Motion and Animation #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.15625

openalex publication_date 2025/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Musicians and nonmusicians alike use rhythmic sound gestures, such as tapping and beatboxing, to express drum patterns. While these gestures effectively communicate musical ideas, realizing these ideas as fully-produced drum recordings can be time-consuming, potentially disrupting many creative workflows. To bridge this gap, we present TRIA (The Rhythm In Anything), a masked transformer model for mapping rhythmic sound gestures to high-fidelity drum recordings. Given an audio prompt of the desired rhythmic pattern and a second prompt to represent drumkit timbre, TRIA produces audio of a drumkit playing the desired rhythm (with appropriate elaborations) in the desired timbre. Subjective and objective evaluations show that a TRIA model trained on less than 10 hours of publicly-available drum data can generate high-quality, faithful realizations of sound gestures across a wide range of timbres in a zero-shot manner.

Citations

• Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
• KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation
• High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching
• ReNoise: Real Image Inversion Through Iterative Noising
• StemGen: A music generation model that listens
• Moisesdb: A dataset for source separation beyond 4-stems
• VampNet: Music Generation via Masked Acoustic Token Modeling
• High-Fidelity Audio Compression with Improved RVQGAN
• SoundStorm: Efficient Parallel Audio Generation
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Classifier-Free Diffusion Guidance
• MaskGIT: Masked Generative Image Transformer
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
• BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Tracking
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
• Comparing Representations for Audio Synthesis Using Generative Adversarial Networks
• DDSP: Differentiable Digital Signal Processing
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interaction

Related

• Setting the rhythm scene: deep learning-based drum loop generation from arbitrary language cues
• JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAE
• Language Models are Drummers: Drum Composition with Natural Language Pre-Training
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• MaskBeat: Loopable Drum Beat Generation