2024/12/02 by Jiang, RuiZhe, Haotian Lei, Lei, Haotian
Engineering · #Advanced Control Systems Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reservoir Engineering and Simulation Methods #Topology Optimization in Engineering
paper · pdf · doi:10.48550/arxiv.2412.01476
openalex publication_date 2024/12/02 · openalex created_date 2024/12/05 · openalex updated_date 2026/07/28
Over-parameterized neural network models often lead to significant performance discrepancies between training and test sets, a phenomenon known as overfitting. To address this, researchers have proposed numerous regularization techniques tailored to various tasks and model architectures. In this paper, we introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. Based on this viewpoint, we propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set. Due to minimal prior assumptions, this approach is applicable to almost any architecture and task. Our experiments show that it effectively reduces overfitting, with low sensitivity to hyperparameters and minimal computational cost. It demonstrates particularly strong memory suppression and promotes normal convergence, even when the model has already started to overfit. Even in the absence of significant overfitting, our method consistently improves accuracy and reduces validation loss.