Sample size selection in optimization methods for machine learning

2012/06/23 by Richard H. Byrd, Gillian M. Chin, Jorge Nocedal +1 · 42 citations
Engineering · Mathematics · Computer Science · #Sparse and Compressive Sensing Techniques #Numerical methods in inverse problems #Stochastic Gradient Optimization Techniques

paper · doi:10.1007/s10107-012-0572-5

Citations

• A Stochastic Approximation Method
• De-noising by soft-thresholding
• Primal-dual subgradient methods for convex problems

Cited by

• Automated machine learning for pollen-based reconstructions: Evaluation with modern pollen dataset from China and application to Oligocene records
• Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients
• A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Randomized Approach to Nonlinear Inversion Combining Simultaneous Random\n and Optimized Sources and Detectors
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
• TensorFlow: A system for large-scale machine learning
• Convergence-Guaranteed Algorithms for l1/2-Regularized Quadratic Programs with Assignment Constraints
• BISTRO -- A Bi-Fidelity Stochastic Gradient Framework using Trust-Regions for Optimization Under Uncertainty
• Global convergence rate analysis of unconstrained optimization methods\n based on probabilistic models
• A Second-Order Method for Convex \ℓ₁-Regularized Optimization with\n Active Set Prediction
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Training Neural Networks with Stochastic Hessian-Free Optimization
• A Progressive Batching L-BFGS Method for Machine Learning
• Convergence of trust-region methods based on probabilistic models
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Advances in Variational Inference
• Mini-batch stochastic gradient descent with dynamic sample sizes
• Large batch size training of neural networks with adversarial training and second-order information
• Adaptive Sampling Quasi-Newton Methods for Derivative-Free Stochastic\n Optimization
• A line search framework with restarting for noisy optimization problems
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Stochastic Adaptive Gradient Descent Without Descent
• Extragradient method with variance reduction for stochastic variational inequalities
• Stable architectures for deep neural networks
• AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
• A subspace-accelerated split Bregman method for sparse data recovery\n with joint l1-type regularizers
• Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Boosting Accelerated Proximal Gradient Method with Adaptive Sampling for Stochastic Composite Optimization
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• Coupling Adaptive Batch Sizes with Learning Rates
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction
• Optimization Methods for Large-Scale Machine Learning
• A polynomial expansion line search for large-scale unconstrained\n minimization of smooth L2-regularized loss functions, with implementation in\n Apache Spark
• A Dynamic Sampling Adaptive-SGD Method for Machine Learning
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks