Teach Me Sign: Stepwise Prompting LLM for Sign Language Production

2025/07/15 by Zhaoyi An, An, Zhaoyi, Rei Kawakami +1
Computer Science · Health Professions · Psychology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Interpreting and Communication in Healthcare #Multimedia (cs.MM)

paper · pdf · doi:10.48550/arxiv.2507.10972

openalex publication_date 2025/07/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models, with their strong reasoning ability and rich knowledge, have brought revolution to many tasks of AI, but their impact on sign language generation remains limited due to its complexity and unique rules. In this paper, we propose TEAch Me Sign (TEAM-Sign), treating sign language as another natural language. By fine-tuning an LLM, we enable it to learn the correspondence between text and sign language, and facilitate generation. Considering the differences between sign and spoken language, we employ a stepwise prompting strategy to extract the inherent sign language knowledge within the LLM, thereby supporting the learning and generation process. Experimental results on How2Sign and Phoenix14T datasets demonstrate that our approach effectively leverages both the sign language knowledge and reasoning capabilities of LLM to align the different distribution and grammatical rules between sign and spoken language.

Citations

• GPT-4o System Card
• The Llama 3 Herd of Models
• Qwen2 Technical Report
• Vision language models are blind: Failing to translate detailed visual features into words
• T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text
• Neural Sign Actors: A diffusion model for 3D sign language production from text
• LLM-grounded Video Diffusion Models
• Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
• Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining
• MotionGPT: Human Motion as a Foreign Language
• MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
• SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation
• Visual Instruction Tuning
• LLaMA: Open and Efficient Foundation Language Models
• Ham2Pose: Animating Sign Language Notation into Pose Sequences
• Changing the Representation: Examining Language Representation for Neural Sign Language Production
• LoRA: Low-Rank Adaptation of Large Language Models
• How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language
• Language Models are Few-Shot Learners
• Progressive Transformers for End-to-End Sign Language Production
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• Neural Discrete Representation Learning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation
• SignLLM: Sign Language Production Large Language Models
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator
• OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages
• Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios