2024/09/21 by Xin Jiang, Junwei Zheng, Jiang, Xin +11 · 2 citations
Engineering · Health Professions · Social Sciences · #Assistive Technology in Communication and Mobility #Computer Vision and Pattern Recognition (cs.CV) #Digital Accessibility for Disabilities #FOS: Computer and information sciences #Smart Cities and Technologies
paper · pdf · doi:10.48550/arxiv.2409.14215
openalex publication_date 2024/09/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
As Vision-Language Models (VLMs) advance, human-centered Assistive Technologies (ATs) for helping People with Visual Impairments (PVIs) are evolving into generalists, capable of performing multiple tasks simultaneously. However, benchmarking VLMs for ATs remains under-explored. To bridge this gap, we first create a novel AT benchmark (@Bench). Guided by a pre-design user study with PVIs, our benchmark includes the five most crucial vision-language tasks: Panoptic Segmentation, Depth Estimation, Optical Character Recognition (OCR), Image Captioning, and Visual Question Answering (VQA). Besides, we propose a novel AT model (@Model) that addresses all tasks simultaneously and can be expanded to more assistive functions for helping PVIs. Our framework exhibits outstanding performance across tasks by integrating multi-modal information, and it offers PVIs a more comprehensive assistance. Extensive experiments prove the effectiveness and generalizability of our framework.