Federated learning, ethics, and the double black box problem in medical AI
2025/04/29 by Hatherley, Joshua, Søgaard, Anders, Ballantyne, Angela +1
#Artificial Intelligence (cs.AI) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2504.20656
Abstract
Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive double black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.
Citations
- Implementing a Nordic-Baltic Federated Health Data Network: a case report
- Recent Methodological Advances in Federated Learning for Healthcare
- Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models
- Federated Learning of Medical Concepts Embedding using BEHRT
- Efficient Classification of SARS-CoV-2 Spike Sequences Using Federated Learning
- The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default
- Vertical Federated Learning: Concepts, Advances, and Challenges
- Incentive Mechanisms for Federated Learning: From Economic and Game Theoretic Perspective
- Federated Learning for Smart Healthcare: A Survey
- Multi-task Federated Learning for Heterogeneous Pancreas Segmentation
- Defining the undefinable: the black box problem in healthcare artificial intelligence
- Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edge
- Automated Pancreas Segmentation Using Multi-institutional Collaborative Deep Learning
- Transparency in Complex Computational Systems
- Clinical AI: opacity, accountability, responsibility and liability
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
- Limits of trust in medical AI
- The future of digital health with federated learning
- Adaptive Federated Optimization
- Estimating Training Data Influence by Tracing Gradient Descent
- Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results
- High Hopes for “Deep Medicine”? AI, Economics, and the Future of Care
- Advances and Open Problems in Federated Learning
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning
- Federated Learning: Challenges, Methods, and Future Directions
- FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
- Differential Privacy Has Disparate Impact on Model Accuracy
- BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning
- Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records
- Adversarial attacks on medical machine learning
- Agnostic Federated Learning
- Federated Optimization in Heterogeneous Networks
- Representer Point Selection for Explaining Deep Neural Networks
- Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation
- A Survey Of Methods For Explaining Black Box Models
- Understanding Black-box Predictions via Influence Functions
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- The philosophical novelty of computer simulation methods
Related