Activation-Descent Regularization for Input Optimization of ReLU Networks

2024/06/01 by Hongzhan Yu, Yu, Hongzhan, Sicun Gao +1
Computer Science · #Artificial Intelligence (cs.AI) #Energy Efficient Wireless Sensor Networks #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2406.00494

openalex publication_date 2024/06/01 · openalex created_date 2024/06/06 · openalex updated_date 2026/07/28

Abstract

We present a new approach for input optimization of ReLU networks that explicitly takes into account the effect of changes in activation patterns. We analyze local optimization steps in both the input space and the space of activation patterns to propose methods with superior local descent properties. To accomplish this, we convert the discrete space of activation patterns into differentiable representations and propose regularization terms that improve each descent step. Our experiments demonstrate the effectiveness of the proposed input-optimization methods for improving the state-of-the-art in various areas, such as adversarial learning, generative modeling, and reinforcement learning.

Related

• Learning ReLU Networks via Alternating Minimization
• Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU\n Activations
• Optimizing Objective Functions from Trained ReLU Neural Networks via Sampling
• Deep ReLU Programming