2021/10/14 by Alfonso Landeros, Kenneth Lange, Landeros, Alfonso +1
Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning and Algorithms #Methodology (stat.ME) #Sparse and Compressive Sensing Techniques #stat.ME
paper · pdf · doi:10.48550/arxiv.2110.07691
Main text: 21 pages, 3 figures, 4 tables; Appendix: 6 pages, 2 figures
arxiv created 2021/10/14 · openalex publication_date 2021/10/14 · arxiv updated 2021/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Many problems in classification involve huge numbers of irrelevant features. Model selection reveals the crucial features, reduces the dimensionality of feature space, and improves model interpretation. In the support vector machine literature, model selection is achieved by ℓ1 penalties. These convex relaxations seriously bias parameter estimates toward 0 and tend to admit too many irrelevant features. The current paper presents an alternative that replaces penalties by sparse-set constraints. Penalties still appear, but serve a different purpose. The proximal distance principle takes a loss function L(\boldsymbolβ) and adds the penalty \fracρ2dist(\boldsymbolβ, Sk)2 capturing the squared Euclidean distance of the parameter vector \boldsymbolβ to the sparsity set Sk where at most k components of \boldsymbolβ are nonzero. If \boldsymbolβρ represents the minimum of the objective fρ(\boldsymbolβ)=L(\boldsymbolβ)+\fracρ2dist(\boldsymbolβ, Sk)2, then \boldsymbolβρ tends to the constrained minimum of L(\boldsymbolβ) over Sk as ρ tends to ∞. We derive two closely related algorithms to carry out this strategy. Our simulated and real examples vividly demonstrate how the algorithms achieve much better sparsity without loss of classification power.