Reusing Weights in Subword-aware Neural Language Models

2018/02/23 by Assylbekov, Zhenisbek, Takhanov, Rustem
#68T50 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.1802.08375

Abstract

We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-aware model, but some of them improve the performance of syllable- and morpheme-aware models while showing significant reductions in model sizes. We discover a simple hands-on principle: in a multi-layer input embedding model, layers should be tied consecutively bottom-up if reused at output. Our best morpheme-aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%-87% fewer parameters.

Related

• Using the Output Embedding to Improve Language Models
• Patterns versus Characters in Subword-aware Neural Language Modeling
• Adaptive Input Representations for Neural Language Modeling
• Lightweight Adaptation of Neural Language Models via Subspace Embedding
• Syllable-level Neural Language Model for Agglutinative Language