Spoken question answering for visual queries

2025/05/29 by Nimrod Shabtay, Shabtay, Nimrod, Zvi Kons +9 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Multimodal Machine Learning Applications #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.23308

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy.

Citations

• Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
• TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
• What matters when building vision-language models?
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
• AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs
• Improved Baselines with Visual Instruction Tuning
• Progressive Spatio-temporal Perception for Audio-Visual Question Answering
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• HeySQuAD: A Spoken Question Answering Dataset
• Visual Instruction Tuning
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• TVLT: Textless Vision-Language Transformer
• Learning to Answer Questions in Dynamic Audio-Visual Scenarios
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• Knowledge Distillation for Improved Accuracy in Spoken Question Answering
• DocVQA: A Dataset for VQA on Document Images
• ODSQA: Open-domain Spoken Question Answering Dataset
• Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension
• Speech-Based Visual Question Answering
• VQA: Visual Question Answering

Cited by

• TextlessRAG: End-to-End Visual Document RAG by Speech Without Text

Related

• Survey of Recent Advances in Visual Question Answering
• Speech-Based Visual Question Answering
• Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model
• Tackling VQA with Pretrained Foundation Models without Further Training
• SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering