2021/12/07 by Rob van der Goot, Miryam de Lhoneux, van der Goot, Rob +1
Computer Science · #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2112.03625
Accepted to TLT at SyntaxFest 2021
arxiv created 2021/12/07 · arxiv updated 2021/12/08
With an increase of dataset availability, the potential for learning from a variety of data sources has increased. One particular method to improve learning from multiple data sources is to embed the data source during training. This allows the model to learn generalizable features as well as distinguishing features between datasets. However, these dataset embeddings have mostly been used before contextualized transformer-based embeddings were introduced in the field of Natural Language Processing. In this work, we compare two methods to embed datasets in a transformer-based multilingual dependency parser, and perform an extensive evaluation. We show that: 1) embedding the dataset is still beneficial with these models 2) performance increases are highest when embedding the dataset at the encoder level 3) unsurprisingly, we confirm that performance increases are highest for small datasets and datasets with a low baseline score. 4) we show that training on the combination of all datasets performs similarly to designing smaller clusters based on language-relatedness.