Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures

2020/05/05 by Zein Shaheen, Shaheen, Zein, Gerhard Wohlgenannt +7
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2005.02470

openalex publication_date 2020/05/05 · openalex created_date 2020/05/13 · openalex updated_date 2026/07/28

Abstract

Generating coherent, grammatically correct, and meaningful text is very challenging, however, it is crucial to many modern NLP systems. So far, research has mostly focused on English language, for other languages both standardized datasets, as well as experiments with state-of-the-art models, are rare. In this work, we i) provide a novel reference dataset for Russian language modeling, ii) experiment with popular modern methods for text generation, namely variational autoencoders, and generative adversarial networks, which we trained on the new dataset. We evaluate the generated text regarding metrics such as perplexity, grammatical correctness and lexical diversity.

Citations

• Auto-Encoding Variational Bayes
• Pointer Sentinel Mixture Models
• GANS for Sequences of Discrete Elements with the Gumbel-softmax\n Distribution
• MaskGAN: Better Text Generation via Filling in the______
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

Related

• GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
• Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
• Texts in, meaning out: neural language models in semantic similarity task for Russian
• Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
• From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation