Cost-Sensitive Approach to Batch Size Adaptation for Gradient Descent

2017/12/09 by Matteo Pirotta, Pirotta, Matteo, Marcello Restelli +1
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and ELM #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1712.03428

openalex publication_date 2017/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a novel approach to automatically determine the batch size in stochastic gradient descent methods. The choice of the batch size induces a trade-off between the accuracy of the gradient estimate and the cost in terms of samples of each update. We propose to determine the batch size by optimizing the ratio between a lower bound to a linear or quadratic Taylor approximation of the expected improvement and the number of samples used to estimate the gradient. The performance of the proposed approach is empirically compared with related methods on popular classification tasks. The work was presented at the NIPS workshop on Optimizing the Optimizers. Barcelona, Spain, 2016.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Gradient-based learning applied to document recognition

Related

• On the Batch Size Selection in Stochastic Gradient Methods Using No-Replacement Sampling
• Coupling Adaptive Batch Sizes with Learning Rates
• Improving the convergence of SGD through adaptive batch sizes
• Big Batch SGD: Automated Inference using Adaptive Batch Sizes
• AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size