Conveying Meaning through Gestures: An Investigation into Semantic Co-Speech Gesture Generation

2025/10/20 by Hendric Voß, Voss, Hendric, Lisa Michelle Bohnenkamp +3
Computer Science · Psychology · #Action Observation and Synchronization #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Human-Computer Interaction (cs.HC)

paper · pdf · doi:10.48550/arxiv.2510.17599

openalex publication_date 2025/10/20 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28

Abstract

This study explores two frameworks for co-speech gesture generation, AQ-GT and its semantically-augmented variant AQ-GT-a, to evaluate their ability to convey meaning through gestures and how humans perceive the resulting movements. Using sentences from the SAGA spatial communication corpus, contextually similar sentences, and novel movement-focused sentences, we conducted a user-centered evaluation of concept recognition and human-likeness. Results revealed a nuanced relationship between semantic annotations and performance. The original AQ-GT framework, lacking explicit semantic input, was surprisingly more effective at conveying concepts within its training domain. Conversely, the AQ-GT-a framework demonstrated better generalization, particularly for representing shape and size in novel contexts. While participants rated gestures from AQ-GT-a as more expressive and helpful, they did not perceive them as more human-like. These findings suggest that explicit semantic enrichment does not guarantee improved gesture generation and that its effectiveness is highly dependent on the context, indicating a potential trade-off between specialization and generalization.

Citations

• Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesis
• AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesis
• ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech
• ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech
• Taming Transformers for High-Resolution Image Synthesis
• Speech gesture generation from the trimodal context of text, audio, and speaker identity
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Wasserstein Divergence for GANs
• Neural Discrete Representation Learning
• Generative Adversarial Networks: An Overview
• Attention Is All You Need
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesis
• SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
• SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain
• Large Language Models for Virtual Human Gesture Selection
• ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input