The Efficiency Misnomer

2021/10/25 by Mostafa Dehghani, Anurag Arnab, Dehghani, Mostafa +7 · 5 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2110.12894

openalex publication_date 2021/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

Citations

• ImageNet classification with deep convolutional neural networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Rethinking floating point for deep learning
• Parameter-Efficient Transfer Learning for NLP
• The State of Sparsity in Deep Neural Networks
• Model-Based Reinforcement Learning for Atari
• Lookahead Optimizer: k steps forward, 1 step back
• Scaling Laws for Neural Language Models
• The Cost of Training NLP Models: A Concise Overview
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Long Range Arena: A Benchmark for Efficient Transformers
• Challenges in Deploying Machine Learning: A Survey of Case Studies
• Parameter-Efficient Transfer Learning with Diff Pruning
• DeLighT: Deep and Light-weight Transformer
• Scaling Laws for Transfer
• OmniNet: Omnidirectional Representations from Transformers
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Multiscale Vision Transformers
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Efficient Nearest Neighbor Language Models
• Primer: Searching for Efficient Transformers for Language Modeling
• ResNet strikes back: An improved training procedure in timm

Cited by

• Batched Training for QLSTM vs. QFWP: A System-Oriented Approach to EPC-Aware RMSE-DA
• PerfMamba: Performance Analysis and Pruning of Selective State Space Models
• Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation
• GUIDE: Guided Initialization and Distillation of Embeddings
• Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts

Related

• Rethinking and Recomputing the Value of Machine Learning Models
• Cost-of-Pass: An Economic Framework for Evaluating Language Models
• Not quite unreasonable effectiveness of machine learning algorithms
• Striving for data-model efficiency: Identifying data externalities on group performance
• Is it worth it? Budget-related evaluation metrics for model selection