Training Data is More Valuable than You Think: A Simple and Effective\n Method by Retrieving from Training Data

2022/03/16 by Shuohang Wang, Wang, Shuohang, Xu Yi‐chong +13 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2203.08773

openalex publication_date 2022/03/16 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28

Abstract

Retrieval-based methods have been shown to be effective in NLP tasks via\nintroducing external knowledge. However, the indexing and retrieving of\nlarge-scale corpora bring considerable computational cost. Surprisingly, we\nfound that REtrieving from the traINing datA (REINA) only can lead to\nsignificant gains on multiple NLG and NLU tasks. We retrieve the labeled\ntraining instances most similar to the input text and then concatenate them\nwith the input to feed into the model to generate the output. Experimental\nresults show that this simple method can achieve significantly better\nperformance on a variety of NLU and NLG tasks, including summarization, machine\ntranslation, language modeling, and question answering tasks. For instance, our\nproposed method achieved state-of-the-art results on XSum, BigPatent, and\nCommonsenseQA. Our code is released, https://github.com/microsoft/REINA .\n

Cited by

• Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning
• SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuning
• RE²: Improving Chinese Grammatical Error Correction via Retrieving Appropriate Examples with Explanation
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• Adversarial Demonstration Learning for Low-resource NER Using Dual Similarity
• Large Language Models are Demonstration Pre-Selectors for Themselves

Related

• Few-shot Mining of Naturally Occurring Inputs and Outputs
• D4: Improving LLM Pretraining via Document De-Duplication and Diversification
• Post-training an LLM for RAG? Train on Self-Generated Demonstrations
• REALM: Retrieval-Augmented Language Model Pre-Training
• Making Large Language Models Better Data Creators