2024/02/02 by María José de la Pascua Sánchez, Laura Fernández, Sánchez, María +14
Arts and Humanities · #Audio and Speech Processing (eess.AS) #Cinema History and Criticism #FOS: Computer and information sciences #FOS: Electrical engineering #Photographic and Visual Arts #Sound (cs.SD) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2402.01385
openalex publication_date 2024/02/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent advances in image, video, text and audio generative techniques, and their use by the general public, are leading to new forms of content generation. Usually, each modality was approached separately, which poses limitations. The automatic sound recording of visual sequences is one of the greatest challenges for the automatic generation of multimodal content. We present a processing flow that, starting from images extracted from videos, is able to sound them. We work with pre-trained models that employ complex encoders, contrastive learning, and multiple modalities, allowing complex representations of the sequences for their sonorization. The proposed scheme proposes different possibilities for audio mapping and text guidance. We evaluated the scheme on a dataset of frames extracted from a commercial video game and sounds extracted from the Freesound platform. Subjective tests have evidenced that the proposed scheme is able to generate and assign audios automatically and conveniently to images. Moreover, it adapts well to user preferences, and the proposed objective metrics show a high correlation with the subjective ratings.