Large Margin Neural Language Model

2018/08/27 by Huang, Jiaji, Li, Yi, Ping, Wei +1 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1808.08987

Abstract

We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.

Cited by

• From Injection to Defense: Constructing Edit-Based Fingerprints for Large Language Models

Related

• Investigation of Large-Margin Softmax in Neural Language Modeling
• Strategies for Training Large Vocabulary Neural Language Models
• Pragmatic Neural Language Modelling in Machine Translation
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• Methods for Estimating and Improving Robustness of Language Models