2024/12/11 by Hugo Flores García, García, Hugo Flores, Oriol Nieto +7 · 4 voices · 13 citations
Computer Science · #Acoustics #Audio signal #Computer science #Music Technology and Sound Studies #Music and Audio Processing #Physics #Speech and Audio Processing #Speech coding #Speech recognition
paper · pdf · doi:10.48550/arxiv.2412.08550
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/12/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch, as well as text prompts. Sketch2Sound can synthesize arbitrary sounds from sonic imitations (i.e.,~a vocal imitation or a reference sound-shape). Sketch2Sound can be implemented on top of any text-to-audio latent diffusion transformer (DiT), and requires only 40k steps of fine-tuning and a single linear layer per control, making it more lightweight than existing methods like ControlNet. To synthesize from sketchlike sonic imitations, we propose applying random median filters to the control signals during training, allowing Sketch2Sound to be prompted using controls with flexible levels of temporal specificity. We show that Sketch2Sound can synthesize sounds that follow the gist of input controls from a vocal imitation while retaining the adherence to an input text prompt and audio quality compared to a text-only baseline. Sketch2Sound allows sound artists to create sounds with the semantic flexibility of text prompts and the expressivity and precision of a sonic gesture or vocal imitation. Sound examples are available at https://hugofloresgarcia.art/sketch2sound/.