Some like it tough: Improving model generalization via progressively\n increasing the training difficulty

2021/10/25 by Hannes Fassold, Fassold, Hannes
Computer Science · #Neural Networks and Applications #Machine Learning and Data Classification #Machine Learning and ELM

paper · pdf · doi:10.48550/arxiv.2110.13058

Abstract

In this work, we propose to progressively increase the training difficulty\nduring learning a neural network model via a novel strategy which we call\nmini-batch trimming. This strategy makes sure that the optimizer puts its focus\nin the later training stages on the more difficult samples, which we identify\nas the ones with the highest loss in the current mini-batch. The strategy is\nvery easy to integrate into an existing training pipeline and does not\nnecessitate a change of the network model. Experiments on several image\nclassification problems show that mini-batch trimming is able to increase the\ngeneralization ability (measured via final test error) of the trained model.\n

Related

• Data optimization for large batch distributed training of deep neural\n networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Training Deep Neural Networks Without Batch Normalization
• SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization
• Simplifying Neural Network Training Under Class Imbalance