Neural Networks with Complex-Valued Weights Have No Spurious Local Minima

2021/01/31 by Xingtu Liu, Liu, Xingtu
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2103.07287

openalex publication_date 2021/01/31 · openalex created_date 2021/03/29 · openalex updated_date 2026/07/28

Abstract

We study the benefits of complex-valued weights for neural networks. We prove that shallow complex neural networks with quadratic activations have no spurious local minima. In contrast, shallow real neural networks with quadratic activations have infinitely many spurious local minima under the same conditions. In addition, we provide specific examples to demonstrate that complex-valued weights turn poor local minima into saddle points.

Citations

• Wirtinger's Calculus in general Hilbert Spaces
• The Complex Gradient Operator and the CR-Calculus
• Associative Long Short-Term Memory
• On Complex Valued Convolutional Neural Networks
• Full-Capacity Unitary Recurrent Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Phase-aware Speech Enhancement with Deep Complex U-Net
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• Network size and weights size for memorization with two-layers neural networks
• Optimization Landscapes of Wide Deep Neural Networks Are Benign

Related

• Small nonlinearities in activation functions create bad local minima in neural networks
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks
• The loss surface of deep and wide neural networks