Video-Guided Foley Sound Generation with Multimodal Controls

2024/11/26 by Ziyang Chen, Chen, Ziyang, Prem Seetharaman +12 · 4 voices · 17 citations
Computer Science · #Speech and Audio Processing #cs.CV #cs.MM #cs.SD #eess.AS

paper · pdf · doi:10.48550/arxiv.2411.17698

openalex publication_date 2024/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generating sound effects for videos often requires creating artistic sound effects that diverge significantly from real-life sources and flexible control in the sound design. To address this problem, we introduce MultiFoley, a model designed for video-guided sound generation that supports multimodal conditioning through text, audio, and video. Given a silent video and a text prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels spinning without wind noise) or more whimsical sounds (e.g., making a lion's roar sound like a cat's meow). MultiFoley also allows users to choose reference audio from sound effects (SFX) libraries or partial videos for conditioning. A key novelty of our model lies in its joint training on both internet video datasets with low-quality audio and professional SFX recordings, enabling high-quality, full-bandwidth (48kHz) audio generation. Through automated evaluations and human studies, we demonstrate that MultiFoley successfully generates synchronized high-quality sounds across varied conditional inputs and outperforms existing methods. Please see our project page for video results: https://ificl.github.io/MultiFoley/

Cited by

• ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation
• DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
• ReflexFlow: Rethinking Learning Objective for Exposure Bias Alleviation in Flow Matching
• Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
• VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning
• FoleyBench: A Benchmark For Video-to-Audio Models
• Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
• Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
• SoundReactor: Frame-level Online Video-to-Audio Generation
• Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
• SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos
• Video Object Segmentation-Aware Audio Generation
• VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
• Guiding Audio Editing with Audio Language Model
• LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapters
• VGGSounder: Audio-Visual Evaluations for Foundation Models
• Sonify Anything: Towards Context-Aware Sonic Interactions in AR

Discussions

• We generate a soundtrack for a silent video, given a text prompt! For example, we can make a cat's meow sound like a lion's roar or a typewriter sound like a piano. Paper: arxiv.org/abs/2411.17698 W [bsky, 42 points, 0 comments]
• 🎥 Introducing MultiFoley, a video-aware audio generation method with multimodal controls! 🔊 We can ⌨️Make a typewriter sound like a piano 🎹 🐱Make a cat meow like a lion roars! 🦁 ⏱️Perfectly time [bsky, 42 points, 2 comments]
• Link to research: arXiv: arxiv.org/abs/2411.17698 website: ificl.github.io/MultiFoley/ [bsky, 2 points, 0 comments]
• MultiFoley by Adobe Research MultiFoley is an AI model generating high-quality sound effects from text, audio, and video inputs. Cool demos highlight its creative potential. arxiv.org/abs/2411.17698 [bsky, 0 points, 1 comments]

Related

• YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls
• FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
• Read, Watch and Scream! Sound Generation from Text and Video
• FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment
• Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis