Cross-regularization: Adaptive Model Complexity through Validation Gradients

2025/06/24 by Carlos Stein Naves de Brito, Brito, Carlos Stein
Physics and Astronomy · Computer Science · Engineering · #Model Reduction and Neural Networks #Machine Learning and Algorithms #Reservoir Engineering and Simulation Methods

paper · pdf · doi:10.48550/arxiv.2506.19755

Abstract

Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-validation optima. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specific regularization that emerges organically during training. Beyond complexity control, the framework integrates seamlessly with data augmentation, uncertainty calibration and growing datasets while maintaining single-run efficiency through a simple gradient-based approach.

Citations

• Weighted Training for Cross-Task Learning
• Understanding deep learning (still) requires rethinking generalization
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Deep Double Descent: Where Bigger Models and More Data Hurt
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Population Based Training of Neural Networks
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• Spectrally-normalized margin bounds for neural networks
• On Calibration of Modern Neural Networks
• Concrete Dropout
• Variational Dropout Sparsifies Deep Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Weight Uncertainty in Neural Networks
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Least angle regression

Related

• ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization
• Deep Bilevel Learning
• The Effects of Regularization and Data Augmentation are Class Dependent
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Regularization Path of Cross-Validation Error Lower Bounds