2025/07/07 by Taewan Kim, Jiwoo Kang
Computer Science · #Face and Expression Recognition #Face recognition and analysis #Speech and Audio Processing #cs.CV #cs.IR
paper · pdf · doi:10.1007/s00530-025-01872-9
published as Multimedia Systems, vol. 31, no. 4, pp. 296, July 2025
openalex publication_date 2025/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23 · arxiv created 2026/07/30 · arxiv updated 2026/07/31
Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal attention mechanisms with convex embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.