2021/06/20 by Runqi Wang, Wang, Runqi, Baochang Zhang +8
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #cs.LG
paper · pdf · doi:10.48550/arxiv.2106.10617
arXiv admin note: substantial text overlap with arXiv:2006.09142
arxiv created 2021/06/20 · openalex publication_date 2021/06/20 · arxiv updated 2021/06/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Conventional gradient descent methods compute the gradients for multiple variables through the partial derivative. Treating the coupled variables independently while ignoring the interaction, however, leads to an insufficient optimization for bilinear models. In this paper, we propose a dependable learning based on Cogradient Descent (CoGD) algorithm to address the bilinear optimization problem, providing a systematic way to coordinate the gradients of coupling variables based on a kernelized projection function. CoGD is introduced to solve bilinear problems when one variable is with sparsity constraint, as often occurs in modern learning paradigms. CoGD can also be used to decompose the association of features and weights, which further generalizes our method to better train convolutional neural networks (CNNs) and improve the model capacity. CoGD is applied in representative bilinear problems, including image reconstruction, image inpainting, network pruning and CNN training. Extensive experiments show that CoGD improves the state-of-the-arts by significant margins. Code is available at https://github.com/bczhangbczhang/CoGD.