eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations

2025/09/17 by Lennart Bamberg, Bamberg, Lennart, Filippo Minnella +13 · 2 citations
Computer Science · #Advanced Neural Network Applications #Parallel Computing and Optimization Techniques #Big Data and Digital Economy

paper · pdf · doi:10.48550/arxiv.2509.14388

Abstract

Neural Processing Units (NPUs) are key to enabling efficient AI inference in resource-constrained edge environments. While peak tera operations per second (TOPS) is often used to gauge performance, it poorly reflects real-world performance and typically rather correlates with higher silicon cost. To address this, architects must focus on maximizing compute utilization, without sacrificing flexibility. This paper presents the eIQ Neutron efficient-NPU, integrated into a commercial flagship MPU, alongside co-designed compiler algorithms. The architecture employs a flexible, data-driven design, while the compiler uses a constrained programming approach to optimize compute and data movement based on workload characteristics. Compared to the leading embedded NPU and compiler stack, our solution achieves an average speedup of 1.8x (4x peak) at equal TOPS and memory resources across standard AI-benchmarks. Even against NPUs with double the compute and memory resources, Neutron delivers up to 3.3x higher performance.

Citations

• Stream: Design Space Exploration of Layer-Fused DNNs on Heterogeneous Dataflow Accelerators
• DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modeling
• DAMO-YOLO : A Report on Real-Time Object Detection Design
• FP8 Quantization: The Power of the Exponent
• CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• MCUNet: Tiny Deep Learning on IoT Devices
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• EfficientDet: Scalable and Efficient Object Detection
• EfficientDet: Scalable and Efficient Object Detection
• Searching for MobileNetV3
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Microsoft COCO: Common Objects in Context

Cited by

• On-Device Inference versus Wireless Streaming: Energy-Efficient Multi-Modal Deep Learning for Wearable Cardiovascular Patches
• Rescaling-Aware Training for Efficient Deployment of Deep Learning Models on Full-Integer Hardware

Related

• Benchmarking Edge AI Platforms for High-Performance ML Inference
• Exploiting Neural-Network Statistics for Low-Power DNN Inference
• Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends
• Bit-Line Computing for CNN Accelerators Co-Design in Edge AI Inference
• Hardware-Aware Neural Network Compilation with Learned Optimization: A RISC-V Accelerator Approach