Model Complexity of Deep Learning: A Survey

2021/03/08 by Xia Hu, Lingyang Chu, Hu, Xia +7 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2103.05127

openalex publication_date 2021/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.

Citations

• Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition
• DARTS: Differentiable Architecture Search

Cited by

• Hybrid modeling approach for better identification of building thermal network model and improved prediction
• On Optimal Hyperparameters for Differentially Private Deep Transfer Learning
• A novel Information-Driven Strategy for Optimal Regression Assessment
• Generating High-Quality Datasets for Code Editing via Open-Source Language Models
• FAWN: A MultiEncoder Fusion-Attention Wave Network for Integrated Sensing and Communication Indoor Scene Inference
• Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunities

Related

• A Two-Scale Complexity Measure for Deep Learning Models
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
• Models, networks and algorithmic complexity
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey