Extracting speaker and emotion information from self-supervised speech models via channel-wise correlations

2022/10/15 by Stafylakis, Themos, Mosner, Ladislav, Kakouros, Sofoklis +3 · 2 citations
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2210.09513

Abstract

Self-supervised learning of speech representations from large amounts of unlabeled data has enabled state-of-the-art results in several speech processing tasks. Aggregating these speech representations across time is typically approached by using descriptive statistics, and in particular, using the first- and second-order statistics of representation coefficients. In this paper, we examine an alternative way of extracting speaker and emotion information from self-supervised trained models, based on the correlations between the coefficients of the representations - correlation pooling. We show improvements over mean pooling and further gains when the pooling methods are combined via fusion. The code is available at github.com/Lamomal/s3prl_correlation.

Cited by

• Sounding Like a Winner? Prosodic Differences in Post-Match Interviews
• Investigating the Impact of Word Informativeness on Speech Emotion Recognition

Related

• Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing
• Representation learning through cross-modal conditional teacher-student\n training for speech emotion recognition
• Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis
• Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective
• Layer-wise Analysis of a Self-supervised Speech Representation Model