2024/12/13 by George Arthur Baker, Baker, George Arthur, Ankush Raut +7 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2412.10079
openalex publication_date 2024/12/13 · openalex created_date 2024/12/17 · openalex updated_date 2026/07/28
Previous work finds that recent long-context language models fail to make\nequal use of information in the middle of their inputs, preferring pieces of\ninformation located at the tail ends which creates an undue bias in situations\nwhere we would like models to be equally capable of using different parts of\nthe input. Thus far, the problem has mainly only been considered in settings\nwith single pieces of critical information, leading us to question what happens\nwhen multiple necessary pieces of information are spread out over the inputs.\nHere, we demonstrate the effects of the "lost in the middle" problem in the\nmulti-hop question answering setting -- in which multiple reasoning "hops" over\ndisconnected documents are required -- and show that performance degrades not\nonly with respect to the distance of information from the edges of the context,\nbut also between pieces of information. Additionally, we experiment with means\nof alleviating the problem by reducing superfluous document contents through\nknowledge graph triple extraction and summarization, and prompting models to\nreason more thoroughly using chain-of-thought prompting.\n