Convergence rates of deep ReLU networks for multiclass classification

2021/08/02 by Thijs Bos, Bos, Thijs, Johannes Schmidt-Hieber +1 · 3 citations
Computer Science · #68T07 #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning and Algorithms #Primary: 62G05 #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques #secondary: 63H30

paper · pdf · doi:10.48550/arxiv.2108.00969

openalex publication_date 2021/08/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For classification problems, trained deep neural networks return probabilities of class memberships. In this work we study convergence of the learned probabilities to the true conditional class probabilities. More specifically we consider sparse deep ReLU network reconstructions minimizing cross-entropy loss in the multiclass classification setup. Interesting phenomena occur when the class membership probabilities are close to zero. Convergence rates are derived that depend on the near-zero behaviour via a margin-type condition.

Cited by

• A Two-Sample Test of Text Generation Similarity
• Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphs
• Super-fast rates of convergence for Neural Networks Classifiers under the Hard Margin Condition

Related

• Fast convergence rates of deep neural networks for classification
• Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes
• Optimal Convergence Rates of Deep Neural Network Classifiers
• Sharp Rate of Convergence for Deep Neural Network Classifiers under the Teacher-Student Setting
• Convergence of Deep ReLU Networks