Convexification of Neural Graph

2018/01/09 by Han Xiao, Xiao, Han
Computer Science · Mathematics · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1801.02901

openalex publication_date 2018/01/09 · arxiv created 2018/01/13 · arxiv updated 2018/01/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Traditionally, most complex intelligence architectures are extremely non-convex, which could not be well performed by convex optimization. However, this paper decomposes complex structures into three types of nodes: operators, algorithms and functions. Iteratively, propagating from node to node along edge, we prove that "regarding the tree-structured neural graph, it is nearly convex in each variable, when the other variables are fixed." In fact, the non-convex properties stem from circles and functions, which could be transformed to be convex with our proposed scale mechanism. Experimentally, we justify our theoretical analysis by two practical applications.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Gradient-based learning applied to document recognition
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Hungarian Layer: Logics Empowered Neural Architecture
• NDT: Neual Decision Tree Towards Fully Functioned Neural Graph
• Bilateral Multi-Perspective Matching for Natural Language Sentences

Related

• What makes a neural code convex?
• The Curious Case of Convex Neural Networks
• Tightening convex relaxations of trained neural networks: a unified approach for convex and S-shaped activations
• Asymptotic convexity of wide and shallow neural networks
• On convex decision regions in deep network representations