Multi-modal Multi-label Facial Action Unit Detection with Transformer

2022/03/24 by Lingfeng Wang, Wang, Lingfeng, Shisen Wang +3
Computer Science · Psychology · #Computer Vision and Pattern Recognition (cs.CV) #Emotion and Mood Recognition #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #cs.CV

paper · pdf · doi:10.48550/arxiv.2203.13301

openalex publication_date 2022/03/24 · arxiv created 2022/03/28 · arxiv updated 2022/03/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Facial Action Coding System is an important approach of facial expression analysis.This paper describes our submission to the third Affective Behavior Analysis (ABAW) 2022 competition. We proposed a transfomer based model to detect facial action unit (FAU) in video. To be specific, we firstly trained a multi-modal model to extract both audio and visual feature. After that, we proposed a action units correlation module to learn relationships between each action unit labels and refine action unit detection result. Experimental results on validation dataset shows that our method achieves better performance than baseline model, which verifies that the effectiveness of proposed network.

Related

• Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild
• An Attention-based Method for Action Unit Detection at the 3rd ABAW Competition
• Vision Transformer for Action Units Detection
• Transformer-based Multimodal Information Fusion for Facial Expression Analysis
• Multi-label Transformer for Action Unit Detection