Input-to-Output Gate to Improve RNN Language Models

2017/09/26 by Takase, Sho, Suzuki, Jun, Nagata, Masaaki
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1709.08907

Abstract

This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.

Related

• Gated Feedback Recurrent Neural Networks
• Direct Output Connection for a High-Rank Language Model
• Gates Are Not What You Need in RNNs
• Revisiting Activation Regularization for Language RNNs
• HGRN2: Gated Linear RNNs with State Expansion