2019/04/12 by Adrian Bulat, Bulat, Adrian, Jean Kossaifi +6 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #Computer science #Deep learning #Domain (mathematical analysis) #Forgetting #Machine learning #Mathematics #Rank (graph theory) #Set (abstract data type) #Task (project management) #Tensor (intrinsic definition) #Tensor decomposition and applications #cs.AI #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.1904.06345
published in arXiv (Cornell University) (Cornell University) · AAAI20
openalex publication_date 2019/04/12 · arxiv created 2019/11/22 · arxiv updated 2019/11/25 · openalex created_date 2022/07/29 · openalex updated_date 2026/08/06
The prominence of deep learning, large amount of annotated data and\nincreasingly powerful hardware made it possible to reach remarkable performance\nfor supervised classification tasks, in many cases saturating the training\nsets. However the resulting models are specialized to a single very specific\ntask and domain. Adapting the learned classification to new domains is a hard\nproblem due to at least three reasons: (1) the new domains and the tasks might\nbe drastically different; (2) there might be very limited amount of annotated\ndata on the new domain and (3) full training of a new model for each new task\nis prohibitive in terms of computation and memory, due to the sheer number of\nparameters of deep CNNs. In this paper, we present a method to learn\nnew-domains and tasks incrementally, building on prior knowledge from already\nlearned tasks and without catastrophic forgetting. We do so by jointly\nparametrizing weights across layers using low-rank Tucker structure. The core\nis task agnostic while a set of task specific factors are learnt on each new\ndomain. We show that leveraging tensor structure enables better performance\nthan simply using matrix operations. Joint tensor modelling also naturally\nleverages correlations across different layers. Compared with previous methods\nwhich have focused on adapting each layer separately, our approach results in\nmore compact representations for each new task/domain. We apply the proposed\nmethod to the 10 datasets of the Visual Decathlon Challenge and show that our\nmethod offers on average about 7.5x reduction in number of parameters and\ncompetitive performance in terms of both classification accuracy and Decathlon\nscore.\n