Return of the RNN: Residual Recurrent Networks for Invertible Sentence Embeddings

2023/03/23 by Jeremy Wilkerson, Wilkerson, Jeremy
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2303.13570

openalex publication_date 2023/03/23 · openalex created_date 2023/03/30 · openalex updated_date 2026/07/28

Abstract

This study presents a novel model for invertible sentence embeddings using a residual recurrent network trained on an unsupervised encoding task. Rather than the probabilistic outputs common to neural machine translation models, our approach employs a regression-based output layer to reconstruct the input sequence's word vectors. The model achieves high accuracy and fast training with the ADAM optimizer, a significant finding given that RNNs typically require memory units, such as LSTMs, or second-order optimization methods. We incorporate residual connections and introduce a "match drop" technique, where gradients are calculated only for incorrect words. Our approach demonstrates potential for various natural language processing applications, particularly in neural network-based systems that require high-quality sentence embeddings.

Related

• Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs
• Backward and Forward Language Modeling for Constrained Sentence Generation
• Recurrent Neural Network Grammars
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems