Meaningful Pose-Based Sign Language Evaluation

2025/10/08 by Zifan Jiang, Colin Leong, Jiang, Zifan +19
Computer Science · Engineering · Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Human Motion and Animation

paper · pdf · doi:10.48550/arxiv.2510.07453

openalex publication_date 2025/10/08 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level retrieval and a human correlation study of text-to-pose translation across different sign languages. Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign language translation or generation systems.

Citations

• Modelling the Distribution of Human Motion for Sign Language Assessment
• T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text
• A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars
• Neural Sign Actors: A diffusion model for 3D sign language production from text
• SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark
• pose-format: Library for Viewing, Augmenting, and Handling .pose Files
• Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining
• YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus
• Improving 3D Pose Estimation for Sign Language
• An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation
• DUB: Discrete Unit Back-translation for Speech Translation
• Considerations for meaningful sign language machine translation based on glosses
• Ham2Pose: Animating Sign Language Notation into Pose Sequences
• COMET-QE and Active Learning for Low-Resource Machine Translation
• Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
• Sockeye 3: Fast Neural Machine Translation with PyTorch
• MotionCLIP: Exposing Human Motion Generation to CLIP Space
• Improving Sign Language Translation with Monolingual Data by Sign Back-Translation
• Data Augmentation for Sign Language Gloss Translation
• Including Signed Languages in Natural Language Processing
• Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks
• Deep Learning-Based Human Pose Estimation: A Survey
• Deep Learning-based Human Pose Estimation: A Survey
• How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign\n Language
• Adversarial Training for Multi-Channel Sign Language Production
• Progressive Transformers for End-to-End Sign Language Production
• BLEURT: Learning Robust Metrics for Text Generation
• Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
• MediaPipe: A Framework for Building Perception Pipelines
• BERTScore: Evaluating Text Generation with BERT
• Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
• A Structured Review of the Validity of BLEU
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator

Related

• Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insights
• Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognition
• Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibility
• Ham2Pose: Animating Sign Language Notation into Pose Sequences
• An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface