2021/11/30 by Jérôme Darbon, Darbon, Jérôme, Gabriel P. Langlois +1
Computer Science · Engineering · Mathematics · #49M29 #62J12 #65K10 #FOS: Computer and information sciences #FOS: Mathematics #G.1.6 #I.2.6 #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #acm:49M29 #acm:62J12 #acm:65K10 #cs.LG #math.OC #msc:49M29 #msc:62J12 #msc:65K10
paper · pdf · doi:10.48550/arxiv.2111.15426
15 pages
openalex publication_date 2021/11/30 · arxiv created 2021/12/28 · arxiv updated 2021/12/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Logistic regression is a widely used statistical model to describe the relationship between a binary response variable and predictor variables in data sets. It is often used in machine learning to identify important predictor variables. This task, variable selection, typically amounts to fitting a logistic regression model regularized by a convex combination of ℓ1 and ℓ22 penalties. Since modern big data sets can contain hundreds of thousands to billions of predictor variables, variable selection methods depend on efficient and robust optimization algorithms to perform well. State-of-the-art algorithms for variable selection, however, were not traditionally designed to handle big data sets; they either scale poorly in size or are prone to produce unreliable numerical results. It therefore remains challenging to perform variable selection on big data sets without access to adequate and costly computational resources. In this paper, we propose a nonlinear primal-dual algorithm that addresses these shortcomings. Specifically, we propose an iterative algorithm that provably computes a solution to a logistic regression problem regularized by an elastic net penalty in O(T(m,n)log(1/ε)) operations, where ε∈ (0,1) denotes the tolerance and T(m,n) denotes the number of arithmetic operations required to perform matrix-vector multiplication on a data set with m samples each comprising n features. This result improves on the known complexity bound of O(min(m2n,mn2)log(1/ε)) for first-order optimization methods such as the classic primal-dual hybrid gradient or forward-backward splitting methods.