2019/06/07 by Ognjen Rudovic, Rudovic, Ognjen, Meiru Zhang +5
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics #Robotics (cs.RO) #Social Robot Interaction and HRI
paper · pdf · doi:10.48550/arxiv.1906.03098
openalex publication_date 2019/06/07 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
Human behavior expression and experience are inherently multi-modal, and\ncharacterized by vast individual and contextual heterogeneity. To achieve\nmeaningful human-computer and human-robot interactions, multi-modal models of\nthe users states (e.g., engagement) are therefore needed. Most of the existing\nworks that try to build classifiers for the users states assume that the data\nto train the models are fully labeled. Nevertheless, data labeling is costly\nand tedious, and also prone to subjective interpretations by the human coders.\nThis is even more pronounced when the data are multi-modal (e.g., some users\nare more expressive with their facial expressions, some with their voice).\nThus, building models that can accurately estimate the users states during an\ninteraction is challenging. To tackle this, we propose a novel multi-modal\nactive learning (AL) approach that uses the notion of deep reinforcement\nlearning (RL) to find an optimal policy for active selection of the users data,\nneeded to train the target (modality-specific) models. We investigate different\nstrategies for multi-modal data fusion, and show that the proposed model-level\nfusion coupled with RL outperforms the feature-level and modality-specific\nmodels, and the naive AL strategies such as random sampling, and the standard\nheuristics such as uncertainty sampling. We show the benefits of this approach\non the task of engagement estimation from real-world child-robot interactions\nduring an autism therapy. Importantly, we show that the proposed multi-modal AL\napproach can be used to efficiently personalize the engagement classifiers to\nthe target user using a small amount of actively selected users data.\n