LLM2Fx-Tools: Tool Calling For Music Post-Production

2025/12/01 by Seungheon Doh, Doh, Seungheon, Junghyun Koo +11 · 1 citation
Computer Science · #Music and Audio Processing #Music Technology and Sound Studies #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2512.01559

Abstract

This paper introduces LLM2Fx-Tools, a multimodal tool-calling framework that generates executable sequences of audio effects (Fx-chain) for music post-production. LLM2Fx-Tools uses a large language model (LLM) to understand audio inputs, select audio effects types, determine their order, and estimate parameters, guided by chain-of-thought (CoT) planning. We also present LP-Fx, a new instruction-following dataset with structured CoT annotations and tool calls for audio effects modules. Experiments show that LLM2Fx-Tools can infer an Fx-chain and its parameters from pairs of unprocessed and processed audio, enabled by autoregressive sequence modeling, tool calling, and CoT reasoning. We further validate the system in a style transfer setting, where audio effects information is transferred from a reference source and applied to new content. Finally, LLM-as-a-judge evaluation demonstrates that our approach generates appropriate CoT reasoning and responses for music production queries. To our knowledge, this is the first work to apply LLM-based tool calling to audio effects modules, enabling interpretable and controllable music production.

Citations

• TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling
• TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation
• MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures
• ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors
• Reverse Engineering of Music Mixing Graphs with Differentiable Processors and Iterative Pruning
• Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching
• Can Large Language Models Predict Audio Effects Parameters from Natural Language?
• Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior
• Qwen3 Technical Report
• TALKPLAY: Multimodal Music Recommendation with Large Language Models
• CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
• ADIFF: Explaining audio difference using natural language
• ST-ITO: Controlling Audio Effects for Style Transfer with Inference-Time Optimization
• The Llama 3 Herd of Models
• Qwen2-Audio Technical Report
• Diff-MST: Differentiable Mixing Style Transfer
• Searching For Music Mixing Graphs: A Pruning Approach
• Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
• LLark: A Multimodal Instruction-Following Language Model for Music
• Audio Difference Learning for Audio Captioning
• General Purpose Audio Effect Removal
• LP-MusicCaps: LLM-Based Pseudo Music Captioning
• Moisesdb: A dataset for source separation beyond 4-stems
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Visual Instruction Tuning
• Blind Estimation of Audio Processing Graph
• GPT-4 Technical Report
• Toolformer: Language Models Can Teach Themselves to Use Tools
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Toward Universal Text-to-Music Retrieval
• PAL: Program-aided Language Models
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects
• Contrastive Audio-Language Learning for Music
• Automatic music mixing with deep learning and out-of-domain data
• Style Transfer of Audio Effects with Differentiable Signal Processing
• Emergent Abilities of Large Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Differentiable Signal Processing With Black-Box Audio Effects
• Contrastive Learning of Musical Representations
• Exploiting Cloze Questions for Few Shot Text Classification and Natural\n Language Inference
• Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
• A Feature Learning Siamese Model for Intelligent Control of the Dynamic Range Compressor

Cited by

• LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation
• TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling

Related

• Can Large Language Models Predict Audio Effects Parameters from Natural Language?
• SonicRAG : High Fidelity Sound Effects Synthesis Based on Retrival Augmented Generation
• StemFX: Learning Mixing Style Representations via Autoregressive FX Chain Prediction on Source-Separated Stems
• Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing