LLM Augmentations to support Analytical Reasoning over Multiple Documents

2024/11/25 by Raquib Bin Yousuf, Yousuf, Raquib Bin, Nicholas Defelice +7 · 1 citation
Computer Science · #Advanced Database Systems and Queries #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2411.16116

openalex publication_date 2024/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Building on their demonstrated ability to perform a variety of tasks, we investigate the application of large language models (LLMs) to enhance in-depth analytical reasoning within the context of intelligence analysis. Intelligence analysts typically work with massive dossiers to draw connections between seemingly unrelated entities, and uncover adversaries' plans and motives. We explore if and how LLMs can be helpful to analysts for this task and develop an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads. Through extensive experiments on multiple datasets, we highlight how LLMs, as-is, are still inadequate to support intelligence analysts and offer recommendations to improve LLMs for such intricate reasoning applications.

Cited by

• Can an LLM Induce a Graph? Investigating Memory Drift and Context Length

Related

• Translating Expert Intuition into Quantifiable Features: Encode Investigator Domain Knowledge via LLM for Enhanced Predictive Analytics
• An Evaluation of Large Language Models Using Analytic Tradecraft Standards
• LLMs for Multi-Modal Knowledge Extraction and Analysis in Intelligence/Safety-Critical Applications
• Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models
• LLM Inference Enhanced by External Knowledge: A Survey