2021/01/10 by Peter Washington, Onur Cezmi Mutlu, Washington, Peter +19 · 1 citation
Psychology · Computer Science · Medicine · #Emotion and Mood Recognition #Domain Adaptation and Few-Shot Learning #COVID-19 diagnosis using AI
paper · pdf · doi:10.48550/arxiv.2101.03477
Emotion classifiers traditionally predict discrete emotions. However, emotion\nexpressions are often subjective, thus requiring a method to handle subjective\nlabels. We explore the use of crowdsourcing to acquire reliable soft-target\nlabels and evaluate an emotion detection classifier trained with these labels.\nWe center our study on the Child Affective Facial Expression (CAFE) dataset, a\ngold standard collection of images depicting pediatric facial expressions along\nwith 100 human labels per image. To test the feasibility of crowdsourcing to\ngenerate these labels, we used Microworkers to acquire labels for 207 CAFE\nimages. We evaluate both unfiltered workers as well as workers selected through\na short crowd filtration process. We then train two versions of a classifiers\non soft-target CAFE labels using the original 100 annotations provided with the\ndataset: (1) a classifier trained with traditional one-hot encoded labels, and\n(2) a classifier trained with vector labels representing the distribution of\nCAFE annotator responses. We compare the resulting softmax output distributions\nof the two classifiers with a 2-sample independent t-test of L1 distances\nbetween the classifier's output probability distribution and the distribution\nof human labels. While agreement with CAFE is weak for unfiltered crowd\nworkers, the filtered crowd agree with the CAFE labels 100% of the time for\nmany emotions. While the F1-score for a one-hot encoded classifier is much\nhigher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the\noutput probability vector of the crowd-trained classifier more closely\nresembles the distribution of human labels (t=3.2827, p=0.0014). Reporting an\nemotion probability distribution that accounts for the subjectivity of human\ninterpretation. Crowdsourcing, including a sufficient filtering mechanism, is a\nfeasible solution for acquiring soft-target labels.\n