Wav2CLIP: Learning Robust Audio Representations From CLIP

2021/10/21 by Ho-Hsiang Wu, Wu, Ho-Hsiang, Prem Seetharaman +5 · 41 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #cs.LG #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2110.11499

Copyright 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

arxiv created 2022/02/15 · arxiv updated 2022/02/16

Abstract

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.

Cited by

• FlowBind: Efficient Any-to-Any Generation with Bidirectional Flows
• ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval
• AudioScene: Integrating Object-Event Audio into 3D Scenes
• PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning
• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
• Calibrated Multimodal Representation Learning with Missing Modalities
• Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model
• Large language model-based task planning for service robots: A review
• Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards
• MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding
• Latent Multi-view Learning for Robust Environmental Sound Representations
• SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization
• SpeechCT-CLIP: Distilling Text-Image Knowledge to Speech for Voice-Native Multimodal CT Analysis
• Audio-Visual Separation with Hierarchical Fusion and Representation Alignment
• Testing chatbots on the creation of encoders for audio conditioned image generation
• Effectively obtaining acoustic, visual and textual data from videos
• Decoding Visual Neural Representations by Multimodal with Dynamic Balancing
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
• Improving Audio Classification by Transitioning from Zero- to Few-Shot
• ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion
• CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation
• Principled Multimodal Representation Learning
• CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning
• Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives
• Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
• Multimodal Spatial Language Maps for Robot Navigation and Manipulation
• Can Sound Replace Vision in LLaVA With Token Substitution?
• Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization
• MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation
• Leveraging CLIP Encoder for Multimodal Emotion Recognition
• AVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time
• Representation Learning for Semantic Alignment of Language, Audio, and Visual Modalities
• SounDiT: Geo-Contextual Soundscape-to-Landscape Generation
• Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
• Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning
• FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence
• ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval
• MV-Crafter: An Intelligent System for Music-guided Video Generation
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection
• A Survey on Cross-Modal Interaction Between Music and Multimodal Data
• TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis

Related

• Exploring Efficient-Tuned Learning Audio Representation Method from BriVL
• Leveraging Pretrained Image-text Models for Improving Audio-Visual Learning
• SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
• CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos