Exchangeability in Neural Network and its Application to Dynamic Pruning
2025/06/02 by Pu Pu, Pu, Yi Yi +7
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Performance (cs.PF)
paper · pdf · doi:10.48550/arxiv.2506.02210
openalex publication_date 2025/06/02 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/31
Abstract
Modern neural networks (NN) contain an ever-growing number of parameters, substantially increasing the memory and computational cost of inference. Researchers have explored various ways to reduce the inference cost of NNs by reducing the model size before deployment and dynamically pruning the inference computation at runtime. In this work, we present ExPrune, a general, dynamic pruning optimization that enables multi-granularity partial computation on a per-input basis. ExPrune requires no change to the model architecture or the training algorithm. ExPrune is based on our theoretical results that the relationship between certain model parameters and intermediate values can be described by a statistical property called exchangeability. By identifying exchangeable parameters and values in the model, we are able to first partially evaluate the network, analyze the statistics of the partial results, and make pruning decisions on the fly. Because ExPrune is theory grounded, it generalizes across model architectures in different problem domains. We evaluate ExPrune on one computer vision models, one graph model and one language model. ExPrune provides 10.98--17.33% reduction in FLOPs with negligible accuracy drop and 21.61--27.16% reduction in FLOPs with at most 1% accuracy drop. We also demonstrate that ExPrune composes with static magnitude pruning. On models that have been aggressively statically pruned, ExPrune still provides additional 10.24--11.11% reduction in FLOPs with negligible accuracy drop and 13.91--14.39% reduction in FLOPs with at most 1% accuracy drop.
Citations
- MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers
- ResNet and Asymmetric ResNet Weights, for "The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof"
- Compressing Large Language Models using Low Rank and Low Precision Decomposition
- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
- A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
- Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
- Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry
- LLaMA: Open and Efficient Foundation Language Models
- Symmetries, flat minima, and the conserved quantities of gradient flow
- On the Symmetries of Deep Learning Models and their Internal Representations
- Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
- A Survey of Quantization Methods for Efficient Neural Network Inference
- Dynamic Neural Networks: A Survey
- EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
- Knowledge Distillation: A Survey
- Exchangeability, Conformal Prediction, and Rank Tests
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Optuna: A Next-generation Hyperparameter Optimization Framework
- BRUNO: A Deep Recurrent Model for Exchangeable Data
- A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- Deep Sets
- MoleculeNet: A Benchmark for Molecular Machine Learning
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
- BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
- Semi-Supervised Classification with Graph Convolutional Networks
- Layer Normalization
- Deep Residual Learning for Image Recognition
- All you need is a good init
- Learning both Weights and Connections for Efficient Neural Networks
- Distilling the Knowledge in a Neural Network
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Efficient Estimation of Word Representations in Vector Space
Related