Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection

2026/08/05 by Mihailo Ilić, Miloš Savić, Vladimir Kurbalija +3
Computer Science · #cs.LG

paper · pdf

Submitted to the 4th IEEE International Conference on Federated Learning Technologies and Applications (FLTA 2026)

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Attention layers are the backbone of today's most powerful and impactful models. Models with multi-million and billion parameters rely on contextual knowledge provided by attention layers. However, their use goes well beyond just being the core component of large language models. One particularly interesting application is in Memory Augmented Autoencoders (MemAE), specifically for unsupervised representation learning in outlier detection tasks. It was shown that attention helps these models be more effective in centralized learning scenarios. Our work aims to address the lack of specialized aggregation techniques in Federated Learning (FL) when it comes to MemAE models. In this paper we analyze the intricacies of the architecture behind Memory Augmented Autoencoders, and propose novel, guided approaches to effectively aggregate these models in federated scenarios. We demonstrate our approach on non-IID datasets and show that these novel aggregation schemes are more robust when dealing with numerous edge nodes in environments with unbalanced datasets, specifically for unsupervised anomaly detection scenarios. This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.

Citations

• FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data
• A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms
• Personalized Federated Learning via Convex Clustering
• Towards Personalized Federated Learning
• Tax Evasion Risk Management Using a Hybrid Unsupervised Outlier Detection Method
• Deep Learning Anomaly Detection for Cellular IoT with Applications in Smart Logistics
• Privacy and Robustness in Federated Learning: Attacks and Defenses
• A Unifying Review of Deep and Shallow Anomaly Detection
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Advances and Open Problems in Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• apricot: Submodular selection for data summarization in Python
• Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints
• Representation Learning: A Review and New Perspectives
• Representation Learning: A Review and New Perspectives
• Submodular Function Maximization