Scaling ML Products At Startups: A Practitioner's Guide

2023/04/20 by Atul Dhingra, Gaurav Sood, Dhingra, Atul +1
Business, Management and Accounting · Computer Science · #Big Data and Business Intelligence #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2304.10660

openalex publication_date 2023/04/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

How do you scale a machine learning product at a startup? In particular, how do you serve a greater volume, velocity, and variety of queries cost-effectively? We break down costs into variable costs-the cost of serving the model and performant-and fixed costs-the cost of developing and training new models. We propose a framework for conceptualizing these costs, breaking them into finer categories, and limn ways to reduce costs. Lastly, since in our experience, the most expensive fixed cost of a machine learning system is the cost of identifying the root causes of failures and driving continuous improvement, we present a way to conceptualize the issues and share our methodology for the same.

Related

• Costing Generated Runtime Execution Plans for Large-Scale Machine Learning Programs
• The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models
• Training and Serving Machine Learning Models at Scale
• Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review
• Reasonable Scale Machine Learning with Open-Source Metaflow