Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
2022/11/12 by Yusong Wu, Ke Chen, Wu, Yusong +11 · 1 voice · 286 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.SD #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2211.06687
openalex publication_date 2022/11/12 · arxiv published 2022/11/12 · openalex created_date 2022/11/23 · arxiv updated 2024/03/21 · openalex updated_date 2026/07/28
Abstract
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
Cited by
- AudioGAN: A Compact and Efficient Framework for Real-Time High-Fidelity Text-to-Audio Generation
- Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models
- Adaptive Acoustic Monitoring for Endangered Cook Inlet Beluga Whales in Complex Soundscapes
- Text-Prompted CLAP: Learning Query-Conditioned Audio Representations via Contrastive Learning
- M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
- QuarkAudio Technical Report
- Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
- Do Foundational Audio Encoders Understand Music Structure?
- Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal
- JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing
- MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
- BRACE: A Benchmark for Robust Audio Caption Quality Evaluation
- VABench: A Comprehensive Benchmark for Audio-Video Generation
- DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
- Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
- WhAM: Towards A Translative Model of Sperm Whale Vocalization
- LLM2Fx-Tools: Tool Calling For Music Post-Production
- CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
- CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
- Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
- CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries
- EBind: a practical approach to space binding
- Calibrated Multimodal Representation Learning with Missing Modalities
- Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
- Listening Between the Frames: Bridging Temporal Gaps in Large Audio-Language Models
- End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering
- Video Echoed in Music: Semantic, Temporal, and Rhythmic Alignment for Video-to-Music Generation
- Melodia: Training-Free Music Editing Guided by Attention Probing in Diffusion Models
- Automatic Music Mixing using a Generative Model of Effect Embeddings
- DOA Estimation with Lightweight Network on LLM-Aided Simulated Acoustic Scenes
- PromptSep: Generative Audio Separation via Multimodal Prompting
- MusRec: Zero-Shot Text-to-Music Editing via Rectified Flow and Diffusion Transformers
- Tortoise and Hare Guidance: Accelerating Diffusion Model Inference with Multirate Integration
- MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models
- UniTok-Audio: A Unified Audio Generation Framework via Generative Modeling on Discrete Codec Tokens
- FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Video
- SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations
- OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
- Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robots
- Text2Score: Generating Sheet Music From Textual Prompts
- Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
- 'Studies for': A Human-AI Co-Creative Sound Artwork Using a Real-time Multi-channel Sound Generation Model
- MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding
- ETC: training-free diffusion models acceleration with Error-aware Trend Consistency
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
- FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search
- Steering Autoregressive Music Generation with Recursive Feature Machines
- ProLAP: Probabilistic Language-Audio Pre-Training
- AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering
- Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?
- Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module
- MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding
- UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
- Scope: Selective Cross-modal Orchestration of Visual Perception Experts
- DeePAQ: A Perceptual Audio Quality Metric Based On Foundational Models and Weakly Supervised Learning
- Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
- UALM: Unified Audio Language Model for Understanding, Generation and Reasoning
- Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions
- Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap
- MSRBench: A Benchmarking Dataset for Music Source Restoration
- MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
- SoundReactor: Frame-level Online Video-to-Audio Generation
- Multi-bit Audio Watermarking
- Bias beyond Borders: Global Inequalities in AI-Generated Music
- MelTok: 2D Tokenization for Single-Codebook Audio Compression
- SLAP: Learning Speaker and Health-Related Representations from Natural Language Supervision
- MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation
- ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
- Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information
- LARA-Gen: Enabling Continuous Emotion Control for Music Generation Models via Latent Affective Representation Alignment
- FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoders
- StereoSync: Spatially-Aware Stereo Audio Generation from Video
- Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
- TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling
- Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoning
- OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
- Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
- Echoes of Humanity: Exploring the Perceived Humanness of AI Music
- EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition
- Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes
- When Audio Generators Become Good Listeners: Generative Features for Understanding Tasks
- AudioMoG: Guiding Audio Generation with Mixture-of-Guidance
- MDAR: A Multi-scene Dynamic Audio Reasoning Benchmark
- MusicWeaver: Programmable Long-Form Music Generation with Provably Local Editing
- Guiding Audio Editing with Audio Language Model
- SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization
- MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization
- SCORE: Scaling audio generation using Standardized COmposite REwards
- Audio-Visual Separation with Hierarchical Fusion and Representation Alignment
- Improved Robustness in AI-Generated Music Detection
- Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis
- MusiCRS: Benchmarking Audio-Centric Conversational Recommendation
- Towards Evaluating Generative Audio: Insights from Neural Audio Codec Embedding Distances
- FlexSED: Towards Open-Vocabulary Sound Event Detection
- SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filtering
- StereoFoley: Object-Aware Stereo Audio Generation from Video
- MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representation
- Revealing Multimodal Causality with Large Language Models
- Virtual Consistency for Audio Editing
- STAR: Speech-to-Audio Generation via Representation Learning
- FakeSound2: A Benchmark for Explainable and Generalizable Deepfake Sound Detection
- UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching
- A Survey on Evaluation Metrics for Music Generation
- Beyond Video-to-SFX: Video to Audio Synthesis with Environmentally Aware Speech
- Multimodal Learning for Fake News Detection in Short Videos Using Linguistically Verified Data and Heterogeneous Modality Fusion
- The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling
- Sound Separation and Classification with Object and Semantic Guidance
- Exploring How Audio Effects Alter Emotion with Foundation Models
- Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance
- Music4All A+A
- Spatial-CLAP: Learning Spatially-Aware audio--text Embeddings for Multi-Source Conditions
- CLAIP-Emo: Parameter-Efficient Adaptation of Language-supervised models for In-the-Wild Audiovisual Emotion Recognition
- Aligning Audio Captions with Human Preferences
- Hashing-Baseline: Rethinking Hashing in the Age of Pretrained Models
- Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing
- Assessing Data Replication in Symbolic Music via Adapted Structural Similarity Index Measure
- GraphMend: Code Transformations for Fixing Graph Breaks in PyTorch 2
- MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models
- Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMs
- Contrastive timbre representations for musical instrument and synthesizer retrieval
- Cross-Modal Retrieval with Cauchy-Schwarz Divergence
- Acoustic Overspecification in Electronic Dance Music Taxonomy
- HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
- CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio
- Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations
- Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets
- Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos
- PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music
- UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
- DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
- Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper
- TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph
- The AudioMOS Challenge 2025
- PicoAudio2: Temporal Controllable Text-to-Audio Generation with Natural Language Description
- DRASP: A Dual-Resolution Attentive Statistics Pooling Framework for Automatic MOS Prediction
- Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music
- Sound event detection with audio-text models and heterogeneous temporal annotations
- SCAR: A Characterization Scheme for Multi-Modal Dataset
- MotionFlux: Efficient Text-Guided Motion Generation through Rectified Flow Matching and Preference Alignment
- MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
- AudioSet-R: A Refined AudioSet with Multi-Stage LLM Label Reannotation
- LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
- MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
- DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer
- SPANER: Shared Prompt Aligner for Multimodal Semantic Representation
- MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
- Exploring Self-Supervised Audio Models for Generalized Anomalous Sound Detection
- What Matters for Bioacoustic Encoding
- Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style
- OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
- Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations
- Listen through the Sound: Generative Speech Restoration Leveraging Acoustic Context Representation
- QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems
- Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning
- MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows
- A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding
- CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization
- SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing
- Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval
- MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation
- AutoMashup: Automatic Music Mashups Creation
- From Waveforms to Pixels: A Survey on Audio-Visual Segmentation
- Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification
- Implicit Counterfactual Learning for Audio-Visual Segmentation
- Controllable Video-to-Music Generation with Multiple Time-Varying Conditions
- Improving Audio Classification by Transitioning from Zero- to Few-Shot
- ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion
- Closing the Modality Gap for Mixed Modality Search
- CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation
- VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation
- ViRN: Variational Inference and Distribution Trilateration for Long-Tailed Continual Representation Learning
- Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generation
- Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries
- Towards Reliable Objective Evaluation Metrics for Generative Singing Voice Separation Models
- EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
- ASTAR-NTU solution to AudioMOS Challenge 2025 Track1
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
- If open source is to win, it must go public
- FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generation
- Dynamic Parameter Memory: Temporary LoRA-Enhanced LLM for Long-Sequence Emotion Recognition in Conversation
- Assessing the Alignment of Audio Representations with Timbre Similarity Ratings
- Generating Moving 3D Soundscapes with Latent Diffusion Models
- ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning
- 2025 Metrics Bake-off: Listener Ratings and Separated Audio
- EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation
- Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos
- TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Models
- CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning
- DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
- Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures
- MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing
- IdolSongsJp Corpus: A Multi-Singer Song Corpus in the Style of Japanese Idol Groups
- RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio
- Human-CLAP: Human-perception-based contrastive language-audio pretraining
- TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure
- Evaluating Sound Similarity Metrics for Differentiable, Iterative Sound-Matching
- Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance
- ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
- Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
- TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
- Benchmarking Music Generation Models and Metrics via Human Preference Studies
- MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
- Large-Scale Training Data Attribution for Music Generative Models via Unlearning
- USAD: Universal Speech and Audio Representation via Distillation
- GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models
- SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding
- Universal Music Representations? Evaluating Foundation Models on World Music Corpora
- ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors
- Hybrid-Sep: Language-queried audio source separation via pre-trained Model Fusion and Adversarial Diffusion Training
- SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
- Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
- Do Music Preferences Reflect Cultural Values? A Cross-National Analysis Using Music Embedding and World Values Survey
- Video-Guided Text-to-Music Generation Using Public Domain Movie Collections
- LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation
- ViSAGe: Video-to-Spatial Audio Generation
- GLAP: General contrastive audio-text pretraining across domains and languages
- PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs
- Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization
- WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction
- FLAM: Frame-Wise Language-Audio Modeling
- Survey on the Evaluation of Generative Models in Music
- Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification
- DRONEAUDIONET: Noise Suppression for Drone Audition-based Search and Rescue
- CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech
- InfiniteAudio: Infinite-Length Audio Generation with Consistency
- DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
- CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer
- FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion
- IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
- Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions
- EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
- Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates
- ACE-Step: A Step Towards Music Generation Foundation Model
- Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles
- AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
- Patient-Aware Feature Alignment for Robust Lung Sound Classification:Cohesion-Separation and Global Alignment Losses
- Can Large Language Models Predict Audio Effects Parameters from Natural Language?
- Text-Queried Audio Source Separation via Hierarchical Modeling
- Music Source Restoration
- Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks
- RA-CLAP: Relation-Augmented Emotional Speaking Style Contrastive Language-Audio Pretraining For Speech Retrieval
- EnvSDD: Benchmarking Environmental Sound Deepfake Detection
- Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embedding
- Learning Normal Patterns in Musical Loops
- SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
- Music Restoration via Latent Operator Optimization and Diffusion Model Priors
- Towards Pre-training an Effective Respiratory Audio Foundation Model
- AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound
- Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation
- Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping
- Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
- Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio
- Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment
- Audio-Visual Contact Classification for Tree Structures in Agriculture
- Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model
- Discovering and Steering Interpretable Concepts in Large Generative Music Models
- FATE: Frame-Level Audio-Visual Temporal Embedding
- Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild
- T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback
- GlobalMood: A cross-cultural benchmark for music emotion recognition
- UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing
- Fast Text-to-Audio Generation with Adversarial Post-Training
- TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining
- Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
- TeMTG: Text-Enhanced Multi-Hop Temporal Graph Modeling for Audio-Visual Video Parsing
- Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations
- Versatile Framework for Song Generation with Prompt-based Control
- SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
- Unleashing the Power of Natural Audio Featuring Multiple Sound Sources
- DOSE : Drum One-Shot Extraction from Music Mixture
- AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation
- InvFlowFD: Reference-Free and Background-Set-Free Perceptual Music Quality Metric with Flow Matching Inversion
- AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models
- Exploring the User Experience of AI-Assisted Sound Searching Systems for Creative Workflows
- DRAGON: Distributional Rewards Optimize Diffusion Generative Models
- LoopGen: Training-Free Loopable Music Generation
- Transformation of audio embeddings into interpretable, concept-based representations
- A Survey on Cross-Modal Interaction Between Music and Multimodal Data
- KVAE: Family of Tokenizers for Multimodal Generative Models
- Rethinking Automatic Music Mixing as Sequential Stem Blending
- Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
- EchoEdit: Stabilizing Inversion-Free Audio Editing via Optimal Transport Geometry
- SteerMusic: Enhanced Musical Consistency for Zero-shot Text-guided and Personalized Music Editing
- TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis
Discussions
Related