ATLAS: Harnessing retrieval-augmented generation (RAG)

2026/01/01 by James Smithies, Glen Berman, Karaitiana Taiuru +2 · 1 voice
Arts and Humanities · Computer Science · Social Sciences · #Digital Humanities and Scholarship #Mathematics, Computing, and Information Processing #Wikis in Education and Collaboration

paper · pdf · doi:10.1017/cfc.2026.10029

openalex publication_date 2026/01/01 · openalex created_date 2026/06/19 · openalex updated_date 2026/06/19

Abstract

Abstract This paper describes a prototype test harness that has been developed to evaluate the use of large language model retrieval-augmented generation systems for humanities and social science research, with an emphasis on collections held in the galleries, libraries, archives and museum sector. We propose the development of new modes of research software engineering oriented towards the values and principles of those communities, and rigorous modes of software testing grounded in industry best practices.

Citations

• On the Dangers of Stochastic Parrots
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Data Worlds: An Introduction
• A Survey on Evaluation of Large Language Models
• Hallucination is Inevitable: An Innate Limitation of Large Language Models
• Holistic Evaluation of Text-To-Image Models
• A Survey on Evaluation of Large Language Models
• Personality Traits in Large Language Models
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
• The Gradient of Generative AI Release: Methods and Considerations
• Neural Language Models for Nineteenth-Century English
• AI assistants in the archive and the lure of “instant history”

Discussions

• @jamessmithies.bsky.social & I have been experimenting with AI for humanities research. Here's our first paper: doi.org/10.1017/cfc..... It introduces a prototype AI test harness, illustrating a path [bsky, 3 points, 0 comments]

Related

• Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
• Deploying Large Language Models With Retrieval Augmented Generation
• Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
• Seven Failure Points When Engineering a Retrieval Augmented Generation System
• Retrieval-Augmented Generation for Large Language Models: A Survey