DOSA: Differentiable Model-Based One-Loop Search for DNN Accelerators

2025/09/12 by C. Hong, Hong, Charles, Qijing Huang +7 · 4 citations
Engineering · Physics and Astronomy · #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Nuclear reactor physics and engineering #Particle Detector Development and Performance #Radiation Detection and Scintillator Technologies

paper · pdf · doi:10.48550/arxiv.2509.10702

openalex publication_date 2025/09/12 · openalex created_date 2025/10/12 · openalex updated_date 2026/07/28

Abstract

In the hardware design space exploration process, it is critical to optimize both hardware parameters and algorithm-to-hardware mappings. Previous work has largely approached this simultaneous optimization problem by separately exploring the hardware design space and the mapspace - both individually large and highly nonconvex spaces - independently. The resulting combinatorial explosion has created significant difficulties for optimizers. In this paper, we introduce DOSA, which consists of differentiable performance models and a gradient descent-based optimization technique to simultaneously explore both spaces and identify high-performing design points. Experimental results demonstrate that DOSA outperforms random search and Bayesian optimization by 2.80x and 12.59x, respectively, in improving DNN model energy-delay product, given a similar number of samples. We also demonstrate the modularity and flexibility of DOSA by augmenting our analytical model with a learned model, allowing us to optimize buffer sizes and mappings of a real DNN accelerator and attain a 1.82x improvement in energy-delay product.

Citations

• Demystifying Map Space Exploration for NPUs
• Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling
• DiGamma: Domain-aware Genetic Algorithm for HW-Mapping Co-optimization for DNN Accelerators
• Data-Driven Offline Optimization For Architecting Hardware Accelerators
• Generative Adversarial Networks
• NAAS: Neural Accelerator Architecture Search
• CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
• TENET: A Framework for Modeling Tensor Dataflow Based on Relation-centric Notation
• HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation
• Apollo: Transferable Architecture Exploration
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via\n Full-Stack Integration
• Attention Is All You Need
• In-Datacenter Performance Analysis of a Tensor Processing Unit
• Aggregated Residual Transformations for Deep Neural Networks
• TensorFlow: A system for large-scale machine learning
• Deep Residual Learning for Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Application-Driven Architecture Exploration for Cross-Layer Heterogeneous Systems
• WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chip
• PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AI
• Mozart: A Chiplet Ecosystem-Accelerator Codesign Framework for Composable Bespoke Application Specific Integrated Circuits

Related

• DiffAxE: Diffusion-driven Hardware Accelerator Generation and Design Space Exploration
• DANCE: Differentiable Accelerator/Network Co-Exploration
• ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning
• DiffuSE: Cross-Layer Design Space Exploration of DNN Accelerator via Diffusion-Driven Optimization
• RHNAS: Realizable Hardware and Neural Architecture Search