Sounding that Object: Interactive Object-Aware Image to Audio Generation

2025/06/04 by Tingle Li, Baihe Huang, Li, Tingle +15 · 1 citation
Computer Science · #Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimedia (cs.MM) #Music Technology and Sound Studies #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.04214

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generating accurate sounds for complex audio-visual scenes is challenging, especially in the presence of multiple objects and sound sources. In this paper, we propose an \em interactive object-aware audio generation model that grounds sound generation in user-selected visual objects within images. Our method integrates object-centric learning into a conditional latent diffusion model, which learns to associate image regions with their corresponding sounds through multi-modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the \em object level. We theoretically validate that our attention mechanism functionally approximates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving better alignment between objects and their associated sounds. Project page: https://tinglok.netlify.app/files/avobject/

Citations

• MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
• Video-Guided Foley Sound Generation with Multimodal Controls
• OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
• Self-Supervised Audio-Visual Soundscape Stylization
• SAM 2: Segment Anything in Images and Videos
• Stable Audio Open
• FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
• Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language
• SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners
• Synchformer: Efficient Synchronization from Sparse Cues
• Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
• CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
• V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Any-to-Any Generation via Composable Diffusion
• ImageBind: One Embedding Space To Bind Them All
• On Uni-Modal Feature Learning in Supervised Multi-Modal Learning
• Conditional Generation of Audio from Video via Foley Analogies
• Soundini: Sound-Guided Diffusion for Natural Video Editing
• Micrograph segmentations for DDEVD
• Segment Anything
• Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment
• Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
• Audio-Text Models Do Not Yet Leverage Natural Language
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation
• CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
• iQuery: Instruments as Queries for Audio-Visual Sound Separation
• I Hear Your True Colors: Image Guided Audio Generation
• AudioGen: Textually Guided Audio Generation
• Classifier-Free Diffusion Guidance
• Diffsound: Discrete Diffusion Model for Text-to-sound Generation
• Learning Visual Styles from Audio-Visual Associations
• Localizing Visual Sounds the Easy Way
• Visual Acoustic Matching
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Audio-Visual Synchronisation in the wild
• Taming Visually Guided Sound Generation
• Neural Dubber: Dubbing for Videos According to Scripts
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Localizing Visual Sounds the Hard Way
• Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
• Learning Transferable Visual Models From Natural Language Supervision
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Denoising Diffusion Implicit Models
• Self-Supervised Learning of Audio-Visual Objects from Video
• Foley Music: Learning to Generate Music from Videos
• Object-Centric Learning with Slot Attention
• Denoising Diffusion Probabilistic Models
• Telling Left from Right: Learning Spatial Correspondence of Sight and Sound
• Atss-Net: Target Speaker Separation via Attention-based Neural Network
• Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis
• VGGSound: A Large-scale Audio-Visual Dataset
• Audio-Visual Instance Discrimination with Cross-Modal Agreement
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Self-Supervised Audio-Visual Co-Segmentation
• The Sound of Motions
• Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators
• MONet: Unsupervised Scene Decomposition and Representation
• Self-Supervised Generation of Spatial Audio for 360 Video
• Representation Learning with Contrastive Predictive Coding
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
• The Sound of Pixels
• Learning to Separate Object Sounds by Watching Unlabeled Video
• Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Objects that Sound
• Visual to Sound: Generating Natural Sound for Videos in the Wild
• Spectrally-normalized margin bounds for neural networks
• Attention Is All You Need
• Look, Listen and Learn
• Vid2speech: Speech Reconstruction from Silent Video
• Improved Techniques for Training GANs
• Visually Indicated Sounds
• Norm-Based Capacity Control in Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Multilayer perceptrons for classification and regression
• Gotta Hear Them All: Towards Sound Source Aware Audio Generation
• Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions

Cited by

• Video Object Segmentation-Aware Audio Generation

Related

• Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions
• Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation
• Align, Adapt and Inject: Sound-guided Unified Image Generation
• Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment
• Audio-Visual Segmentation