Training and Generating Neural Networks in Compressed Weight Space

2021/12/31 by Kazuki Irie, Jürgen Schmidhuber, Irie, Kazuki +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2112.15545

Presented at ICLR 2021 Workshop on Neural Compression, https://openreview.net/forum?id=qU1EUxdVd_D

arxiv created 2021/12/31 · arxiv updated 2022/01/03

Abstract

The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such approaches. Our goal is to open a discussion on this topic, starting with recurrent neural networks for character-level language modelling whose weight matrices are encoded by the discrete cosine transform. Our fast weight version thereof uses a recurrent neural network to parameterise the compressed weights. We present experimental results on the enwik8 dataset.

Cited by

• Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

Related

• Neural Weight Compression for Language Models
• Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
• Weightless: Lossy Weight Encoding For Deep Neural Network Compression
• Learning Compressed Transforms with Low Displacement Rank
• Learning Useful Representations of Recurrent Neural Network Weight Matrices