A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle

2019/01/28 by Harini Suresh, John V. Guttag · 2 voices · 6 citations
Computer Science · Engineering · Mathematics · Medicine · Psychology · Social Sciences · #Adversarial Robustness in Machine Learning #Artificial Intelligence in Healthcare and Education #Business #Computer science #Data science #Downstream (manufacturing) #Engineering #Ethics and Social Impacts of AI #Harm #Operations management #Psychology #Risk analysis (engineering) #Social psychology #Software deployment #Software engineering #cs.LG #stat.ML

paper · pdf · doi:10.1145/3465416.3483305

published as EAAMO 2021: Equity and Access in Algorithms, Mechanisms, and Optimization

arxiv published 2019/01/28 · openalex publication_date 2021/10/05 · arxiv created 2021/12/01 · arxiv updated 2021/12/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. In doing so, we aim to facilitate more productive and precise communication around these issues, as well as more direct, application-grounded ways to mitigate them.

Citations

• Defining Racial and Ethnic Disparities in Pain Management
• Bias in computer systems
• Certifying and Removing Disparate Impact
• Fairness through awareness
• SMOTE: Synthetic Minority Over-sampling Technique
• The Girl Who Cried Pain: A Bias against Women in the Treatment of Pain
• A Convex Framework for Fair Regression
• Counterfactual Fairness
• On Formalizing Fairness in Prediction with Machine Learning
• Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
• No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
• Probably Approximately Metric-Fair Learning
• InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity
• Model Cards for Model Reporting
• Fairness and Abstraction in Sociotechnical Systems
• Inherent Trade-Offs in the Fair Determination of Risk Scores
• Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
• Characterising Bias in Compressed Models

Cited by

• Position: We Need Responsible, Application-Driven (RAD) AI Research
• A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contexts
• Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations
• IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions
• Tracing the bias loop: AI, cultural heritage and bias-mitigating in practice
• Biases in Data Science Lifecycle

Discussions

• System failures can occur at any or all of the steps throughout the machine learning lifecycle; plus in the design, config, etc of the physical hardware. A factor can, but need not, be the physical m [bsky, 2 points, 1 comments]
• A Framework for Understanding Unintended Consequences of Machine Learning [hn, 2 points, 0 comments]

Related

• The Risks of Machine Learning Systems
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Understanding the Nature of System-Related Issues in Machine Learning Frameworks: An Exploratory Study
• The reanimation of pseudoscience in machine learning and its ethical repercussions
• On the Societal Impact of Machine Learning