2023/09/30 by Lee Kezar, Kezar, Lee, Riley Carlin +9 · 1 citation
Computer Science · Psychology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Hearing Impairment and Communication #Speech and dialogue systems
paper · pdf · doi:10.48550/arxiv.2310.00195
openalex publication_date 2023/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Like speech, signs are composed of discrete, recombinable features called phonemes. Prior work shows that models which can recognize phonemes are better at sign recognition, motivating deeper exploration into strategies for modeling sign language phonemes. In this work, we learn graph convolution networks to recognize the sixteen phoneme "types" found in ASL-LEX 2.0. Specifically, we explore how learning strategies like multi-task and curriculum learning can leverage mutually useful information between phoneme types to facilitate better modeling of sign language phonemes. Results on the Sem-Lex Benchmark show that curriculum learning yields an average accuracy of 87% across all phoneme types, outperforming fine-tuning and multi-task strategies for most phoneme types.