2020/11/13 by Amith Ananthram, Kailash Karthik Saravanakumar, Ananthram, Amith +5 · 3 citations
Computer Science · Engineering · Psychology · #Artificial intelligence #Artificial neural network #Audio and Speech Processing (eess.AS) #Classifier (UML) #Computation and Language (cs.CL) #Computer science #Deep neural networks #Emotion and Mood Recognition #Emotion recognition #FOS: Computer and information sciences #FOS: Electrical engineering #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Modal #Natural language processing #Sentiment Analysis and Opinion Mining #Speech Recognition and Synthesis #Speech recognition #Task (project management) #Transfer of learning #cs.CL #cs.HC #cs.LG #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2011.07065
published in arXiv (Cornell University) (Cornell University) · 11 pages, 7 tables, 2 figures
arxiv created 2020/11/13 · openalex publication_date 2020/11/13 · arxiv updated 2020/11/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Automated emotion detection in speech is a challenging task due to the complex interdependence between words and the manner in which they are spoken. It is made more difficult by the available datasets; their small size and incompatible labeling idiosyncrasies make it hard to build generalizable emotion detection systems. To address these two challenges, we present a multi-modal approach that first transfers learning from related tasks in speech and text to produce robust neural embeddings and then uses these embeddings to train a pLDA classifier that is able to adapt to previously unseen emotions and domains. We begin by training a multilayer TDNN on the task of speaker identification with the VoxCeleb corpora and then fine-tune it on the task of emotion identification with the Crema-D corpus. Using this network, we extract speech embeddings for Crema-D from each of its layers, generate and concatenate text embeddings for the accompanying transcripts using a fine-tuned BERT model and then train an LDA - pLDA classifier on the resulting dense representations. We exhaustively evaluate the predictive power of every component: the TDNN alone, speech embeddings from each of its layers alone, text embeddings alone and every combination thereof. Our best variant, trained on only VoxCeleb and Crema-D and evaluated on IEMOCAP, achieves an EER of 38.05%. Including a portion of IEMOCAP during training produces a 5-fold averaged EER of 25.72% (For comparison, 44.71% of the gold-label annotations include at least one annotator who disagrees).