Geometric Insights into Support Vector Machine Behavior using the KKT Conditions

2017/04/03 by Iain Carmichael, J. S. Marron, Carmichael, Iain +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1704.00767

openalex publication_date 2017/04/03 · arxiv created 2018/10/10 · arxiv updated 2018/10/11 · openalex created_date 2022/09/03 · openalex updated_date 2026/07/28

Abstract

The support vector machine (SVM) is a powerful and widely used classification algorithm. This paper uses the Karush-Kuhn-Tucker conditions to provide rigorous mathematical proof for new insights into the behavior of SVM. These insights provide perhaps unexpected relationships between SVM and two other linear classifiers: the mean difference and the maximal data piling direction. For example, we show that in many cases SVM can be viewed as a cropped version of these classifiers. By carefully exploring these connections we show how SVM tuning behavior is affected by characteristics including: balanced vs. unbalanced classes, low vs. high dimension, separable vs. non-separable data. These results provide further insights into tuning SVM via cross-validation by explaining observed pathological behavior and motivating improved cross-validation methodology. Finally, we also provide new results on the geometry of complete data piling directions in high dimensional space.

Related

• Support vector machine and its bias correction in high-dimension, low-sample-size settings
• Support vector machines and linear regression coincide with very high-dimensional features
• A Characterization of the Combined Effects of Overlap and Imbalance on the SVM Classifier
• Behavioral analysis of support vector machine classifier with Gaussian kernel and imbalanced data
• An Algorithm to perform Covariance-Adjusted Support Vector Classification in Non-Euclidean Spaces