An overview of statistical learning theory

1999/01/01 by Vladimir Vapnik, V.N. Vapnik · 76 citations
Computer Science · #Face and Expression Recognition #Machine Learning and Algorithms #Neural Networks and Applications

paper · doi:10.1109/72.788640

openalex publication_date 1999/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Statistical learning theory was introduced in the late 1960's. Until the 1990's it was a purely theoretical analysis of the problem of function estimation from a given collection of data. In the middle of the 1990's new types of learning algorithms (called support vector machines) based on the developed theory were proposed. This made statistical learning theory not only a tool for the theoretical analysis but also a tool for creating practical algorithms for estimating multidimensional functions. This article presents a very general overview of statistical learning theory including both theoretical and algorithmic aspects of the theory. The goal of this overview is to demonstrate how the abstract learning theory established conditions for generalization which are more general than those discussed in classical statistical paradigms and how the understanding of these conditions inspired new algorithmic approaches to function estimation problems. A more detailed overview of the theory (without proofs) can be found in Vapnik (1995). In Vapnik (1998) one can find detailed description of the theory (including proofs).

Cited by

• Resilient Radio Access Networks: AI and the Unknown Unknowns
• Exploring Scale Shift in Crowd Localization under the Context of Domain Generalization
• Connecting Domains and Contrasting Samples: A Ladder for Domain Generalization
• Mixup Helps Understanding Multimodal Video Better
• On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection
• From Formal Language Theory to Statistical Learning: Finite Observability of Subregular Languages
• Spiking Neural Networks: a theoretical framework for Universal Approximation and training
• CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
• Wasserstein Distributionally Robust Adaptive Covariance Steering
• A Primer on Causal and Statistical Dataset Biases for Fair and Robust Image Analysis
• BM-CL: Bias Mitigation through the lens of Continual Learning
• Exploring Over-stationarization in Deep Learning-based Bus/Tram Arrival Time Prediction: Analysis and Non-stationary Effect Recovery
• Universal Reinforcement Learning in Coalgebras: Asynchronous Stochastic Computation via Conduction
• Peer Effect Estimation in the Presence of Simultaneous Feedback and Unobserved Confounders
• Consensus-Driven Active Model Selection
• Quantum-Informed Machine Learning for Predicting Spatiotemporal Chaos
• Handling Out-of-Distribution Data: A Survey
• A Nonparallel Support Tensor Machine for Binary Classification based Large Margin Distribution and Iterative Optimization
• A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
• One-Shot Induction of Generalized Logical Concepts via Human Guidance
• On the consistency of supervised learning with missing values
• Warped-Linear Models for Time Series Classification
• Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergence
• Advances in Machine Learning: Where Can Quantum Techniques Help?
• Locally Consistent Transductive Information Maximization for Few-Shot Remote Sensing Scene Classification
• Deconfounded Video Moment Retrieval with Causal Intervention
• MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition
• On the Domain Robustness of Contrastive Vision-Language Models
• Transductive Episodic-Wise Adaptive Metric for Few-Shot Learning
• Mathematics for Machine Learning
• Exploring Graph-Transformer Out-of-Distribution Generalization Abilities
• Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization
• Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student
• Partial label learning for automated classification of single-cell transcriptomic profiles
• Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
• Robust Training with Data Augmentation for Medical Imaging Classification
• Time Series Anomaly Detection with label-free Model Selection
• High-expressibility Quantum Neural Networks using only classical resources
• Modeling, diagnostics, optimization, and control of internal combustion engines via modern machine learning techniques: A review and future directions
• Improving Group Robustness on Spurious Correlation via Evidential Alignment
• The Sample Complexity of Parameter-Free Stochastic Convex Optimization
• Neural Complexity Measures
• Reliably Detecting Model Failures in Deployment Without Labels
• Uncertainty quantification and stability of neural operators for prediction of three-dimensional turbulence
• Enhancing Treatment Effect Estimation via Active Learning: A Counterfactual Covering Perspective
• Towards Data-Free Domain Generalization
• Statistical Mechanical Analysis of Neural Network Pruning
• Neighborhood Features Help Detecting Non-Technical Losses in Big Data Sets
• Benders Cut Classification via Support Vector Machines for Solving Two-stage Stochastic Programs
• Machine Learning Methods for the Design and Operation of Liquid Rocket Engines -- Research Activities at the DLR Institute of Space Propulsion
• Towards real time assessment of intramuscular fat content in meat using optical fibre-based optical coherence tomography
• Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
• NeuronTune: Towards Self-Guided Spurious Bias Mitigation
• LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation
• ShortcutProbe: Probing Prediction Shortcuts for Learning Robust Models
• Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques
• Information Science Principles of Machine Learning: A Causal Chain Meta-Framework Based on Formalized Information Mapping
• PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning
• Progressive²: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
• Stability Regularized Cross-Validation
• Closing the AI Knowledge Gap
• Social Theory Should Be a Structural Prior for Agentic AI: A Formal Framework for Multi-Agent Social Systems
• Test-time Correlation Alignment
• Inferring Markov chains: Bayesian estimation, model comparison, entropy rate, and out-of-class modeling
• Deep Neural Networks for Choice Analysis: A Statistical Learning Theory Perspective
• Fairness Beyond Demographics: Optimizing Performance Across Appearance-Based Hidden Cohorts in Medical Imaging
• Learning Data Triage: Linear Decoding Works for Compressive MRI
• RBF neural network center selection based on Fisher ratio class separability measure
• Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels
• Reducing the Energy Cost of Inference via In-sensor Information\n Processing
• Graph-Based Transforms for Video Coding
• Robust Label Propagation on Multiple Networks
• The Challenge of Non-Technical Loss Detection Using Artificial Intelligence: A Survey
• Support vector machine for structural reliability analysis
• Residual U-Net for accurate and efficient prediction of hemodynamics in two-dimensional asymmetric stenosis