ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning

2025/05/22 by Tajamul Ashraf, Ashraf, Tajamul, Mohammed Mohsen Peerzada +13
Computer Science · Social Sciences · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2505.16850

openalex publication_date 2025/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Learning (FL) has emerged as a promising paradigm for collaborative model training while preserving data privacy across decentralized participants. As FL adoption grows, numerous techniques have been proposed to tackle its practical challenges. However, the lack of standardized evaluation across key dimensions hampers systematic progress and fair comparison of FL methods. In this work, we introduce ATR-Bench, a unified framework for analyzing federated learning through three foundational dimensions: Adaptation, Trust, and Reasoning. We provide an in-depth examination of the conceptual foundations, task formulations, and open research challenges associated with each theme. We have extensively benchmarked representative methods and datasets for adaptation to heterogeneous clients and trustworthiness in adversarial or unreliable environments. Due to the lack of reliable metrics and models for reasoning in FL, we only provide literature-driven insights for this dimension. ATR-Bench lays the groundwork for a systematic and holistic evaluation of federated learning with real-world relevance. We will make our complete codebase publicly accessible and a curated repository that continuously tracks new developments and research in the FL literature.

Citations

• Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training
• Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learning
• Personalized Federated Learning via Feature Distribution Adaptation
• FedGCA: Global Consistent Augmentation Based Single-Source Federated Domain Generalization
• Federated Representation Learning in the Under-Parameterized Regime
• Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics
• Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark
• Federated Wasserstein Distance
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges
• A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency
• Fairness and Privacy-Preserving in Federated Learning: A Survey
• DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation
• Federated Domain Generalization: A Survey
• Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions
• GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions
• Fair Federated Medical Image Segmentation via Client Contribution Estimation
• Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
• Meta-Learning Based Knowledge Extrapolation for Temporal Knowledge Graph
• FedFA: Federated Feature Augmentation
• Federated Learning on Non-IID Graphs via Structural Knowledge Sharing
• Dim-Krum: Backdoor-Resistant Federated Learning for NLP with Dimension-wise Krum-Based Aggregation
• Federated Domain Generalization for Image Recognition via Cross-Client Style Transfer
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning
• Federated Learning with Label Distribution Skew via Logits Calibration
• Multi-Level Branched Regularization for Federated Learning
• Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation
• Generalized Federated Learning via Sharpness Aware Minimization
• Test-Time Robust Personalization for Federated Learning
• FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data
• IOP-FL: Inside-Outside Personalization for Federated Medical Image Segmentation
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction
• Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation
• Vertical Federated Learning: Challenges, Methodologies and Experiments
• FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients
• Challenges and Approaches for Mitigating Byzantine Attacks in Federated Learning
• HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Federated Learning with Domain Generalization
• Federated Learning for Smart Healthcare: A Survey
• Federated Learning Based on Dynamic Regularization
• What Do We Mean by Generalization in Federated Learning?
• Collaborative Semantic Aggregation and Calibration for Federated Domain Generalization
• Deep Federated Learning for Autonomous Driving
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• FairFed: Enabling Group Fairness in Federated Learning
• FedProc: Prototypical Contrastive Federated Learning on Non-IID data
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• Federated Learning: Issues in Medical Application
• Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning
• Fed2: Feature-Aligned Federated Learning
• Implicit Gradient Alignment in Distributed and Federated Learning
• LoRA: Low-Rank Adaptation of Large Language Models
• CRFL: Certifiably Robust Federated Learning against Backdoor Attacks
• Federated Learning on Non-IID Data: A Survey
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
• FedProto: Federated Prototype Learning across Heterogeneous Clients
• Federated Learning with Fair Averaging
• Model-Contrastive Federated Learning
• FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space
• Learning Transferable Visual Models From Natural Language Supervision
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Provably Secure Federated Learning against Malicious Clients
• FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping
• Ditto: Fair and Robust Federated Learning Through Personalization
• Privacy and Robustness in Federated Learning: Attacks and Defenses
• KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation
• Mitigating Backdoor Attacks in Federated Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Federated Transfer Learning: concept and applications
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Collaborative Fairness in Federated Learning
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Defending against Backdoors in Federated Learning with Robust Learning\n Rate
• Privacy-Preserving Technology to Help Millions of People: Federated Prediction Model for Stroke Prevention
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Decentralised Learning from Independent Multi-Domain Labels for Person Re-Identification
• Hierarchically Fair Federated Learning
• Survey of Personalization Techniques for Federated Learning
• Threats to Federated Learning: A Survey
• Adaptive Federated Optimization
• FedVision: An Online Visual Object Detection Platform Powered by Federated Learning
• Robust Aggregation for Federated Learning
• Private Federated Learning with Domain Adaptation
• Advances and Open Problems in Federated Learning
• Local Model Poisoning Attacks to Byzantine-Robust Federated Learning
• Can You Really Backdoor Federated Learning?
• Federated Adversarial Domain Adaptation
• Overcoming Forgetting in Federated Learning on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Learning: Challenges, Methods, and Future Directions
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• On the Convergence of FedAvg on Non-IID Data
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Fair Resource Allocation in Federated Learning
• Convolutional Networks with Dense Connectivity
• Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach
• Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Agnostic Federated Learning
• Federated Optimization in Heterogeneous Networks
• Federated Learning for Mobile Keyboard Prediction
• Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation
• Mitigating Sybils in Federated Learning Poisoning
• Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net
• How To Backdoor Federated Learning
• Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Towards Deep Learning Models Resistant to Adversarial Attacks
• On Calibration of Modern Neural Networks
• Attention Is All You Need
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Aggregated Residual Transformations for Deep Neural Networks
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Concrete Problems in AI Safety
• Deep Residual Learning for Image Recognition
• Learning with Symmetric Label Noise: The Importance of Being Unhinged
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Gradient-based learning applied to document recognition

Related

• Trustworthy Federated Learning: A Survey
• Federated Learning in Practice: Reflections and Projections
• Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers
• Advances and Open Problems in Federated Learning
• UnifyFL: Enabling Decentralized Cross-Silo Federated Learning