Tortoise and Hare Guidance: Accelerating Diffusion Model Inference with Multirate Integration

2025/11/06 by Byeonghyun Pak, Lee, Yunghee, Pak, Byeonghyun +4
Computer Science · Medicine · Physics and Astronomy · #Advanced Neuroimaging Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2511.04117

openalex publication_date 2025/11/06 · openalex created_date 2025/11/08 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose Tortoise and Hare Guidance (THG), a training-free strategy that accelerates diffusion sampling while maintaining high-fidelity generation. We demonstrate that the noise estimate and the additional guidance term exhibit markedly different sensitivity to numerical error by reformulating the classifier-free guidance (CFG) ODE as a multirate system of ODEs. Our error-bound analysis shows that the additional guidance branch is more robust to approximation, revealing substantial redundancy that conventional solvers fail to exploit. Building on this insight, THG significantly reduces the computation of the additional guidance: the noise estimate is integrated with the tortoise equation on the original, fine-grained timestep grid, while the additional guidance is integrated with the hare equation only on a coarse grid. We also introduce (i) an error-bound-aware timestep sampler that adaptively selects step sizes and (ii) a guidance-scale scheduler that stabilizes large extrapolation spans. THG reduces the number of function evaluations (NFE) by up to 30% with virtually no loss in generation fidelity (ΔImageReward ≤ 0.032) and outperforms state-of-the-art CFG-based training-free accelerators under identical computation budgets. Our findings highlight the potential of multirate formulations for diffusion solvers, paving the way for real-time high-quality image synthesis without any model retraining. The source code is available at https://github.com/yhlee-add/THG.

Citations

• Enhanced Diffusion Sampling via Extrapolation with Multiple ODE Solutions
• Open-Sora: Democratizing Efficient Video Production for All
• MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
• FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality
• Classifier-Free Guidance is a Predictor-Corrector
• CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
• Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching
• Δ-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers
• Unified Directly Denoising for Both Variance Preserving and Variance Exploding Diffusion Models
• Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
• DeepCache: Accelerating Diffusion Models for Free
• Rethinking FID: Towards a Better Evaluation Metric for Image Generation
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• On Distillation of Guided Diffusion Models
• Classifier-Free Diffusion Guidance
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
• CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
• Video Diffusion Models
• Pseudo Numerical Methods for Diffusion Models on Manifolds
• High-Resolution Image Synthesis with Latent Diffusion Models
• Diffusion Models Beat GANs on Image Synthesis
• Learning Transferable Visual Models From Natural Language Supervision
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Microsoft COCO: Common Objects in Context
• Tweedie’s Formula and Selection Bias

Related

• Diffusion Models without Classifier-free Guidance
• DiffIER: Optimizing Diffusion Models with Iterative Error Reduction
• Guided Path Sampling: Steering Diffusion Models Back on Track with Principled Path Guidance
• Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
• Learn to Guide Your Diffusion Model