Sample size selection in optimization methods for machine learning

2012/06/23 by Richard H. Byrd, Gillian M. Chin, Jorge Nocedal +1 · 396 citations
Computer Science · Engineering · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Computation #Computer science #Function (biology) #Gradient descent #Gradient method #Hessian matrix #Mathematical analysis #Mathematical optimization #Mathematics #Numerical methods in inverse problems #Projection (relational algebra) #Sample (material) #Sample size determination #Sparse and Compressive Sensing Techniques #Statistics #Stochastic Gradient Optimization Techniques #Subspace topology

paper · doi:10.1007/s10107-012-0572-5

published in Mathematical Programming 134(1), 127-155 (Springer Science+Business Media)

openalex publication_date 2012/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/25

Citations

• A Stochastic Approximation Method
• De-noising by soft-thresholding
• Primal-dual subgradient methods for convex problems

Cited by

• Automated machine learning for pollen-based reconstructions: Evaluation with modern pollen dataset from China and application to Oligocene records
• Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients
• A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Randomized Approach to Nonlinear Inversion Combining Simultaneous Random\n and Optimized Sources and Detectors
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
• TensorFlow: A system for large-scale machine learning
• Convergence-Guaranteed Algorithms for l1/2-Regularized Quadratic Programs with Assignment Constraints
• BISTRO -- A Bi-Fidelity Stochastic Gradient Framework using Trust-Regions for Optimization Under Uncertainty
• Global convergence rate analysis of unconstrained optimization methods based on probabilistic models
• A Second-Order Method for Convex ℓ₁-Regularized Optimization with Active Set Prediction
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Training Neural Networks with Stochastic Hessian-Free Optimization
• A Progressive Batching L-BFGS Method for Machine Learning
• Convergence of trust-region methods based on probabilistic models
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Advances in Variational Inference
• Mini-batch stochastic gradient descent with dynamic sample sizes
• Large batch size training of neural networks with adversarial training and second-order information
• Adaptive Sampling Quasi-Newton Methods for Derivative-Free Stochastic Optimization
• A line search framework with restarting for noisy optimization problems
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Stochastic Adaptive Gradient Descent Without Descent
• Extragradient method with variance reduction for stochastic variational inequalities
• Stable Architectures for Deep Neural Networks
• AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
• A subspace-accelerated split Bregman method for sparse data recovery\n with joint l1-type regularizers
• Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Boosting Accelerated Proximal Gradient Method with Adaptive Sampling for Stochastic Composite Optimization
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• Coupling Adaptive Batch Sizes with Learning Rates
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction
• Optimization Methods for Large-Scale Machine Learning
• A polynomial expansion line search for large-scale unconstrained minimization of smooth L2-regularized loss functions, with implementation in Apache Spark
• A Dynamic Sampling Adaptive-SGD Method for Machine Learning
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
• An Inexact Variable Metric Proximal Point Algorithm for Generic Quasi-Newton Acceleration
• Stop Wasting My Gradients: Practical SVRG
• Linear Range in Gradient Descent
• Complexity guarantees for risk-neutral generalized Nash equilibrium problems
• Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization
• Stagewise Enlargement of Batch Size for SGD-based Learning
• ASMOP: Additional sampling stochastic trust region method for multi-objective problems
• A variable dimension sketching strategy for nonlinear least-squares
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
• Algorithms for Kullback-Leibler Approximation of Probability Measures in\n Infinite Dimensions
• Adaptive Stochastic Optimization
• An Adaptive Sample Size Trust-Region Method for Finite-Sum Minimization
• Second-order orthant-based methods with enriched Hessian information for sparse ℓ₁-optimization
• Improving the convergence of SGD through adaptive batch sizes
• Trust-Region Algorithms for Training Responses: Machine Learning Methods\n Using Indefinite Hessian Approximations
• Accelerating Mini-batch SARAH by Step Size Rules
• Asynchronous Schemes for Stochastic and Misspecified Potential Games and Nonconvex Optimization
• An enriched second-order method for nonconvex composite sparse optimization problems
• Convergence Rate Analysis of a Stochastic Trust Region Method via Submartingales
• Dynamic sampling schemes for optimal noise learning under multiple\n nonsmooth constraints
• Training Autoencoders Using Stochastic Hessian-Free Optimization with LSMR
• Artificial intelligence method to classify ophthalmic emergency severity based on symptoms: a validation study. [europepmc]
• Comparison of Chemometric Problems in Food Analysis Using Non-Linear Methods. [europepmc]
• Permutation-based identification of important biomarkers for complex diseases via machine learning models. [europepmc]
• Cyber-Physical System for Environmental Monitoring Based on Deep Learning. [europepmc]
• Adaptive stochastic Gauss-Newton method with optical Monte Carlo for quantitative photoacoustic tomography. [europepmc]
• A deep neural network framework to derive interpretable decision rules for accurate traumatic brain injury identification of infants. [europepmc]
• Utilising AI technique to identify depression risk among doctoral students. [europepmc]
• Multidimensional urban morphology and carbon intensity: Nonlinear drivers in dense cities. [europepmc]