2016/08/07 by Matthias Dorfer, Dorfer, Matthias, Andreas Arzt +3 · 4 citations
Computer Science · #Music and Audio Processing #Music Technology and Sound Studies #Speech and Audio Processing
paper · pdf · doi:10.48550/arxiv.1612.05050
This paper addresses the matching of short music audio snippets to the corresponding pixel location in images of sheet music. A system is presented that simultaneously learns to read notes, listens to music and matches the currently played music to its corresponding notes in the sheet. It consists of an end-to-end multi-modal convolutional neural network that takes as input images of sheet music and spectrograms of the respective audio snippets. It learns to predict, for a given unseen audio snippet (covering approximately one bar of music), the corresponding position in the respective score line. Our results suggest that with the use of (deep) neural networks -- which have proven to be powerful image processing models -- working with sheet music becomes feasible and a promising future research direction.