A Survey on Large-scale Machine Learning

2020/08/10 by Meng Wang, Weijie Fu, Wang, Meng +7 · 7 citations
Computer Science · #Algorithms and Data Compression #FOS: Computer and information sciences #Graph Theory and Algorithms #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2008.03911

openalex publication_date 2020/08/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning can provide deep insights into data, allowing machines to make high-quality predictions and having been widely used in real-world applications, such as text mining, visual classification, and recommender systems. However, most sophisticated machine learning approaches suffer from huge time costs when operating on large-scale data. This issue calls for the need of Large-scale Machine Learning (LML), which aims to learn patterns from big data with comparable performance efficiently. In this paper, we offer a systematic survey on existing LML methods to provide a blueprint for the future developments of this area. We first divide these LML methods according to the ways of improving the scalability: 1) model simplification on computational complexities, 2) optimization approximation on computational efficiency, and 3) computation parallelism on computational capabilities. Then we categorize the methods in each perspective according to their targeted scenarios and introduce representative methods in line with intrinsic strategies. Lastly, we analyze their limitations and discuss potential directions as well as open issues that are promising to address in the future.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Adam: A Method for Stochastic Optimization
• Deep Image: Scaling up Image Recognition
• A Survey on Multi-view Learning
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Lower Bound for the Optimization of Finite Sums
• A randomized algorithm for the decomposition of matrices
• Stochastic gradient boosting
• Quantum Support Vector Machine for Big Data Classification
• ImageNet Large Scale Visual Recognition Challenge
• Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
• A survey of decision tree classifier methodology
• Stochastic Gradient Hamiltonian Monte Carlo
• ImageNet classification with deep convolutional neural networks
• Representation Learning: A Review and New Perspectives
• MapReduce
• Variance Reduction in SGD by Distributed Importance Sampling
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• An overview of gradient descent optimization algorithms
• Quantum machine learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• To understand deep learning we need to understand kernel learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Fast Matrix Factorization with Non-Uniform Weights on Missing Data
• Towards Federated Learning at Scale: System Design
• SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• Efficient Large-Scale Learning of Minimax Risk Classifiers
• Sparse and nonparametric estimation of equations governing dynamical systems with applications to biology
• Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method
• Composite Optimization with Error Feedback: the Dual Averaging Approach
• Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunities
• Hadamard-Riemannian Optimization for Margin-Variance Ensemble
• A Distributed Asynchronous Generalized Momentum Algorithm Without Delay Bounds

Related

• Efficient Machine Learning for Big Data: A Review
• Strategies and Principles of Distributed Machine Learning on Big Data
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Challenges and Applications of Large Language Models
• A Survey of Resource-efficient LLM and Multimodal Foundation Models