When Humans Growl and Birds Speak: High-Fidelity Voice Conversion from Human to Animal and Designed Sounds

2025/05/30 by Kang, Minsu, Lee, Seolhee, Lee, Choonghyeon +1 · 1 citation
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.24336

Abstract

Human to non-human voice conversion (H2NH-VC) transforms human speech into animal or designed vocalizations. Unlike prior studies focused on dog-sounds and 16 or 22.05kHz audio transformation, this work addresses a broader range of non-speech sounds, including natural sounds (lion-roars, birdsongs) and designed voice (synthetic growls). To accomodate generation of diverse non-speech sounds and 44.1kHz high-quality audio transformation, we introduce a preprocessing pipeline and an improved CVAE-based H2NH-VC model, both optimized for human and non-human voices. Experimental results showed that the proposed method outperformed baselines in quality, naturalness, and similarity MOS, achieving effective voice conversion across diverse non-human timbres. Demo samples are available at https://nc-ai.github.io/speech/publications/nonhuman-vc/

Citations

• T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis
• DDSP-SFX: Acoustically-guided sound effects generation with differentiable digital signal processing
• PESTO: Pitch Estimation With Self-Supervised Transposition-Equivariant Objective
• Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation
• A Systematic Exploration of Joint-training for Singing Voice Synthesis
• HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer
• High-Fidelity Audio Compression with Improved RVQGAN
• DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation
• Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• CREPE: A Convolutional Representation for Pitch Estimation
• VAE with a VampPrior
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Least Squares Generative Adversarial Networks
• Autoencoding beyond pixels using a learned similarity metric

Cited by

• CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation

Related

• Designed Vocalizations Dataset: Sound-Designed Human and Animal Voices for Non-human Voice Conversion
• An Initial study on Birdsong Re-synthesis Using Neural Vocoders
• What Needs to be Known in Order to Perform a Meaningful Scientific Comparison Between Animal Communications and Human Spoken Language
• CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
• ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds