UnIT: Scalable Unstructured Inference-Time Pruning for MAC-efficient Neural Inference on MCUs

2025/07/10 by Ashe Neth, Neth, Ashe, Sawinder Kaur +9
Computer Science · Engineering · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2507.07885

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing pruning methods are typically applied during training or compile time and often rely on structured sparsity. While compatible with low-power microcontrollers (MCUs), structured pruning underutilizes the opportunity for fine-grained efficiency on devices without SIMD support or parallel compute. To address these limitations, we introduce UnIT (Unstructured Inference-Time pruning), a lightweight method that dynamically identifies and skips unnecessary multiply-accumulate (MAC) operations during inference, guided by input-specific activation patterns. Unlike structured pruning, UnIT embraces irregular sparsity and does not require retraining or hardware specialization. It transforms pruning decisions into lightweight comparisons, replacing multiplications with threshold checks and approximated divisions. UnIT further optimizes compute by reusing threshold computations across multiple connections and applying layer- and group-specific pruning sensitivity. We present three fast, hardware-friendly division approximations tailored to the capabilities of common embedded platforms. Demonstrated on the MSP430 microcontroller, UnIT achieves 11.02% to 82.03% MAC reduction, 27.30% to 84.19% faster inference, and 27.33% to 84.38% lower energy consumption compared to training-time pruned models, while maintaining accuracy with 0.48-7%. Under domain shift, UnIT matches or exceeds the accuracy of retrained models while requiring significantly fewer MACs. These results establish unstructured inference-time pruning as a viable and practical solution for efficient, retraining-free deployment of deep neural networks on MCUs.

Citations

• Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions
• When to Prune? A Policy towards Early Structural Pruning
• Channel Pruning via Optimal Thresholding
• Learned Threshold Pruning
• Structured Compression by Weight Encryption for Unstructured Pruning and Quantization
• Intelligence Beyond the Edge: Inference on Intermittent Embedded Systems
• Rethinking the Value of Network Pruning
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• Google Speech Commands-Musan test set
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Variational Dropout Sparsifies Deep Neural Networks
• BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Structured Pruning of Deep Convolutional Neural Networks
• Quantized Convolutional Neural Networks for Mobile Devices
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning both Weights and Connections for Efficient Neural Networks
• Zero-bias autoencoders and the benefits of co-adapting features
• The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

Related

• Single Shot Structured Pruning Before Training
• Partition Pruning: Parallelization-Aware Pruning for Deep Neural Networks
• Neural network relief: a pruning algorithm based on neural activity
• Efficient CNN Inference on Ultra-Low-Power MCUs via Saturation-Aware Convolution
• HAPM -- Hardware Aware Pruning Method for CNN hardware accelerators in resource constrained devices