Smooth InfoMax -- Towards Easier Post-Hoc Interpretability

2024/08/23 by Denoodt, Fabian, de Boer, Bart, Oramas, José
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2408.12936

Abstract

We introduce Smooth InfoMax (SIM), a self-supervised representation learning method that incorporates interpretability constraints into the latent representations at different depths of the network. Based on β-VAEs, SIM's architecture consists of probabilistic modules optimized locally with the InfoNCE loss to produce Gaussian-distributed representations regularized toward the standard normal distribution. This creates smooth, well-defined, and better-disentangled latent spaces, enabling easier post-hoc analysis. Evaluated on speech data, SIM preserves the large-scale training benefits of Greedy InfoMax while improving the effectiveness of post-hoc interpretability methods across layers.

Related

• Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax
• The Variational InfoMax AutoEncoder
• Learning Representations by Maximizing Mutual Information in Variational\n Autoencoders
• Learning deep representations by mutual information estimation and maximization
• Challenging margin-based speaker embedding extractors by using the variational information bottleneck