2017/04/26 by Lele Chen, Chen, Lele, Sudhanshu Srivastava +5 · 1 voice · 8 citations
Computer Science · Psychology · #Artificial intelligence #Audio visual #Computer science #Generative Adversarial Networks and Image Synthesis #Generative grammar #Machine learning #Modal #Modality (human–computer interaction) #Multimedia #Music Technology and Sound Studies #Music and Audio Processing #Perception #Psychology #Speech recognition #cs.CV #cs.MM #cs.SD
paper · pdf · doi:10.48550/arxiv.1704.08292
published in arXiv (Cornell University) (Cornell University)
arxiv created 2017/04/26 · openalex publication_date 2017/04/26 · arxiv updated 2017/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Cross-modal audio-visual perception has been a long-lasting topic in psychology and neurology, and various studies have discovered strong correlations in human perception of auditory and visual stimuli. Despite works in computational multimodal modeling, the problem of cross-modal audio-visual generation has not been systematically studied in the literature. In this paper, we make the first attempt to solve this cross-modal generation problem leveraging the power of deep generative adversarial training. Specifically, we use conditional generative adversarial networks to achieve cross-modal audio-visual generation of musical performances. We explore different encoding methods for audio and visual signals, and work on two scenarios: instrument-oriented generation and pose-oriented generation. Being the first to explore this new problem, we compose two new datasets with pairs of images and sounds of musical performances of different instruments. Our experiments using both classification and human evaluations demonstrate that our model has the ability to generate one modality, i.e., audio/visual, from the other modality, i.e., visual/audio, to a good extent. Our experiments on various design choices along with the datasets will facilitate future research in this new problem space.