2019/04/04 by Jean Kossaifi, Adrian Bulat, Kossaifi, Jean +5 · 3 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Tensor decomposition and applications
paper · pdf · doi:10.48550/arxiv.1904.02698
openalex publication_date 2019/04/04 · openalex created_date 2022/07/29 · openalex updated_date 2026/07/28
Recent findings indicate that over-parametrization, while crucial for\nsuccessfully training deep neural networks, also introduces large amounts of\nredundancy. Tensor methods have the potential to efficiently parametrize\nover-complete representations by leveraging this redundancy. In this paper, we\npropose to fully parametrize Convolutional Neural Networks (CNNs) with a single\nhigh-order, low-rank tensor. Previous works on network tensorization have\nfocused on parametrizing individual layers (convolutional or fully connected)\nonly, and perform the tensorization layer-by-layer separately. In contrast, we\npropose to jointly capture the full structure of a neural network by\nparametrizing it with a single high-order tensor, the modes of which represent\neach of the architectural design parameters of the network (e.g. number of\nconvolutional blocks, depth, number of stacks, input features, etc). This\nparametrization allows to regularize the whole network and drastically reduce\nthe number of parameters. Our model is end-to-end trainable and the low-rank\nstructure imposed on the weight tensor acts as an implicit regularization. We\nstudy the case of networks with rich structure, namely Fully Convolutional\nNetworks (FCNs), which we propose to parametrize with a single 8th-order\ntensor. We show that our approach can achieve superior performance with small\ncompression rates, and attain high compression rates with negligible drop in\naccuracy for the challenging task of human pose estimation.\n