2021/12/13 by J. K. L. MacDonald, Macdonald, Jan, Stephan Wäldchen +1
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #FOS: Mathematics #Fault Detection and Control Systems #Machine Learning (cs.LG) #Probability (math.PR) #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2112.06532
openalex publication_date 2021/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We give a complete characterisation of families of probability distributions that are invariant under the action of ReLU neural network layers. The need for such families arises during the training of Bayesian networks or the analysis of trained neural networks, e.g., in the context of uncertainty quantification (UQ) or explainable artificial intelligence (XAI). We prove that no invariant parametrised family of distributions can exist unless at least one of the following three restrictions holds: First, the network layers have a width of one, which is unreasonable for practical neural networks. Second, the probability measures in the family have finite support, which basically amounts to sampling distributions. Third, the parametrisation of the family is not locally Lipschitz continuous, which excludes all computationally feasible families. Finally, we show that these restrictions are individually necessary. For each of the three cases we can construct an invariant family exploiting exactly one of the restrictions but not the other two.