Building Audio-Visual Digital Twins with Smartphones

2025/12/11 by Lan, Zitong, Tang, Yiwei, Wang, Yuhan +3
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Multimedia (cs.MM) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2512.10778

Abstract

Digital twins today are almost entirely visual, overlooking acoustics-a core component of spatial realism and interaction. We introduce AV-Twin, the first practical system that constructs editable audio-visual digital twins using only commodity smartphones. AV-Twin combines mobile RIR capture and a visual-assisted acoustic field model to efficiently reconstruct room acoustics. It further recovers per-surface material properties through differentiable acoustic rendering, enabling users to modify materials, geometry, and layout while automatically updating both audio and visuals. Together, these capabilities establish a practical path toward fully modifiable audio-visual digital twins for real-world environments.

Citations

• Guiding Audio Editing with Audio Language Model
• Data Augmentation Using Neural Acoustic Fields With Retrieval-Augmented Pre-training
• Acoustic Volume Rendering for Neural Impulse Response Fields
• LiDAR-GS:Real-time LiDAR Re-Simulation using Gaussian Splatting
• EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer
• GASP: Gaussian Splatting for Physic-Based Simulations
• 3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
• YOLOv10: Real-Time End-to-End Object Detection
• Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction
• SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes
• AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
• Wireless earbuds for low-cost hearing screening
• AutoMerge: A Framework for Map Assembling and Smoothing in City-scale Environments
• Learning Neural Acoustic Fields
• MeshRIR: A Dataset of Room Impulse Responses on Meshed Grid Points For Evaluating Sound Field Analysis and Synthesis Methods
• Digital Twins: State of the Art Theory and Practice, Challenges, and Open Research Questions
• Localization of Sound Sources in a Room with One Microphone

Related

• Differentiable Room Acoustic Rendering with Multi-View Vision Priors
• SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
• Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse
• Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering
• Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction