Repeat after me: Self-supervised learning of acoustic-to-articulatory mapping by vocal imitation

2022/04/05 by Georges, Marc-Antoine, Diard, Julien, Girin, Laurent +2
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2204.02269

Abstract

We propose a computational model of speech production combining a pre-trained neural articulatory synthesizer able to reproduce complex speech stimuli from a limited set of interpretable articulatory parameters, a DNN-based internal forward model predicting the sensory consequences of articulatory commands, and an internal inverse model based on a recurrent neural network recovering articulatory commands from the acoustic speech input. Both forward and inverse models are jointly trained in a self-supervised way from raw acoustic-only speech data from different speakers. The imitation simulations are evaluated objectively and subjectively and display quite encouraging performances.

Related

• From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
• Self-Supervised Models of Speech Infer Universal Articulatory Kinematics
• Embodied Self-supervised Learning by Coordinated Sampling and Training
• Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models
• Simulating Articulatory Trajectories with Phonological Feature Interpolation