2021/12/22 by Christian Oliva, Oliva, Christian, Luis F. Lago-Fernández +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2112.11776
11 pages, 2 appendix pages
arxiv created 2021/12/22 · openalex publication_date 2021/12/22 · arxiv updated 2021/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modeling problems related to natural language processing. Experiments carried out on different problems show that the addition of this kind of connection to a recurrent network always improves the results, regardless of the architecture and training-specific details. When this idea is introduced into the models that lead the field, the resulting networks achieve a new state-of-the-art perplexity in language modeling problems.