Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Prediction

2025/07/07 by Wang, Jun-You, Su, Li
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Multimedia (cs.MM) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.04776

Abstract

We propose a pre-trained BERT-like model for symbolic music understanding that achieves competitive performance across a wide range of downstream tasks. To achieve this target, we design two novel pre-training objectives, namely token correction and pianoroll prediction. First, we sample a portion of note tokens and corrupt them with a limited amount of noise, and then train the model to denoise the corrupted tokens; second, we also train the model to predict bar-level and local pianoroll-derived representations from the corrupted note tokens. We argue that these objectives guide the model to better learn specific musical knowledge such as pitch intervals. For evaluation, we propose a benchmark that incorporates 12 downstream tasks ranging from chord estimation to symbolic genre classification. Results confirm the effectiveness of the proposed pre-training objectives on downstream tasks.

Citations

• A Comparative Study of Clinical ModernBERT and BioMedical ModernBERT on the DDXPlus Dataset
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
• Music Foundation Model as Generic Booster for Music Downstream Tasks
• Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
• The Llama 3 Herd of Models
• MelodyT5: A Unified Score-to-Score Transformer for Symbolic Music Processing
• ChatMusician: Understanding and Generating Music Intrinsically with LLM
• A Foundation Model for Music Informatics
• Roman Numeral Analysis with Graph Neural Networks: Onset-wise Predictions from Note-wise Features
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
• Stable and low-precision training for large-scale vision-language models
• Symbolic Music Structure Analysis with Graph Representations and Changepoint Detection Methods
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Representation Deficiency in Masked Language Modeling
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• On the Opportunities and Risks of Foundation Models
• EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation
• Codified audio language modeling learns useful representations for music information retrieval
• MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
• SUPERB: Speech processing Universal PERformance Benchmark
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Learning Transferable Visual Models From Natural Language Supervision
• Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
• Deep Learning Scaling is Predictable, Empirically
• Attention Is All You Need
• Let Network Decide What to Learn: Symbolic Music Understanding Model Based on Large-scale Adversarial Pre-training
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability
• TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
• What does BERT learn about prosody?
• How to Fine-Tune BERT for Text Classification?
• Revisiting Token Dropping Strategy in Efficient BERT Pretraining