Non-Uniform Stochastic Average Gradient Method for Training Conditional Random Fields

2015/04/16 by Mark Schmidt, Schmidt, Mark, Reza Babanezhad +9 · 2 citations
Computer Science · #Stochastic Gradient Optimization Techniques #Domain Adaptation and Few-Shot Learning #Gaussian Processes and Bayesian Inference

paper · pdf · doi:10.48550/arxiv.1504.04406

Abstract

We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substantially improves practical performance, and analyze the rate of convergence of the SAGA variant under non-uniform sampling. Our experimental results reveal that our method often significantly outperforms existing methods in terms of the training objective, and performs as well or better than optimally-tuned stochastic gradient methods in terms of test error.

Cited by

• Online Batch Selection for Faster Training of Neural Networks
• Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems

Related

• Training Conditional Random Fields with Natural Gradient Descent
• Adaptive Stochastic Dual Coordinate Ascent for Conditional Random Fields
• Separate Training for Conditional Random Fields Using Co-occurrence Rate Factorization
• A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity
• Stochastic Average Gradient : A Simple Empirical Investigation