Adaptive Accompaniment with ReaLchords

2025/06/17 by Yusong Wu, Tim Cooijmans, Wu, Yusong +23 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Parallel Computing and Optimization Techniques #Reinforcement Learning in Robotics #Sound (cs.SD)

paper · pdf · doi:10.48550/arxiv.2506.14723

openalex publication_date 2025/06/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Jamming requires coordination, anticipation, and collaborative creativity between musicians. Current generative models of music produce expressive output but are not able to generate in an online manner, meaning simultaneously with other musicians (human or otherwise). We propose ReaLchords, an online generative model for improvising chord accompaniment to user melody. We start with an online model pretrained by maximum likelihood, and use reinforcement learning to finetune the model for online use. The finetuning objective leverages both a novel reward model that provides feedback on both harmonic and temporal coherency between melody and chord, and a divergence term that implements a novel type of distillation from a teacher model that can see the future melody. Through quantitative experiments and listening tests, we demonstrate that the resulting model adapts well to unfamiliar input and produce fitting accompaniment. ReaLchords opens the door to live jamming, as well as simultaneous co-creation in other modalities.

Citations

• Affective Coherence Monitoring for Transformer-Based Language Models
• Melody transcription via generative pre-training
• Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation
• Scaling Up Models and Data with t5x and seqio
• Training language models to follow instructions with human feedback
• Learning Transferable Visual Models From Natural Language Supervision
• Human-centric Dialog Training via Offline Reinforcement Learning
• Bach or Mock? A Grading Function for Chorales in the Style of J.S. Bach
• RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Hierarchical Reinforcement Learning for Open-Domain Dialog
• Representation Learning with Contrastive Predictive Coding
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Attention Is All You Need
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

Cited by

• Streaming Generation for Music Accompaniment
• Live Music Models

Related

• RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning
• ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformers
• Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
• Reinforcement Learning Jazz Improvisation: When Music Meets Game Theory
• Real-time jam-session support system