Extrapolation in NLP

2018/05/17 by Mitchell, Jeff, Minervini, Pasquale, Stenetorp, Pontus +1
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1805.06648

Abstract

We argue that extrapolation to examples outside the training space will often be easier for models that capture global structures, rather than just maximise their local fit to the training data. We show that this is true for two popular models: the Decomposable Attention Model and word2vec.

Related

• Location Attention for Extrapolation to Longer Sequences
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Learning Representations that Support Extrapolation
• Neural Data Augmentation via Example Extrapolation
• To Know by the Company Words Keep and What Else Lies in the Vicinity