2020/06/10 by Ruozi Huang, Hu Huang, Huang, Ruozi +10 · 8 citations
Computer Science · Engineering · #Human Pose and Action Recognition #Human Motion and Animation #Music and Audio Processing
paper · pdf · doi:10.48550/arxiv.2006.06119
Dancing to music is one of human's innate abilities since ancient times. In\nmachine learning research, however, synthesizing dance movements from music is\na challenging problem. Recently, researchers synthesize human motion sequences\nthrough autoregressive models like recurrent neural network (RNN). Such an\napproach often generates short sequences due to an accumulation of prediction\nerrors that are fed back into the neural network. This problem becomes even\nmore severe in the long motion sequence generation. Besides, the consistency\nbetween dance and music in terms of style, rhythm and beat is yet to be taken\ninto account during modeling. In this paper, we formalize the music-conditioned\ndance generation as a sequence-to-sequence learning problem and devise a novel\nseq2seq architecture to efficiently process long sequences of music features\nand capture the fine-grained correspondence between music and dance.\nFurthermore, we propose a novel curriculum learning strategy to alleviate error\naccumulation of autoregressive models in long motion sequence generation, which\ngently changes the training process from a fully guided teacher-forcing scheme\nusing the previous ground-truth movements, towards a less guided autoregressive\nscheme mostly using the generated movements instead. Extensive experiments show\nthat our approach significantly outperforms the existing state-of-the-arts on\nautomatic metrics and human evaluation. We also make a demo video to\ndemonstrate the superior performance of our proposed approach at\nhttps://www.youtube.com/watch?v=lmE20MEheZ8.\n