dIR -- Discrete Information Retrieval: Conversational Search over Unstructured (and Structured) Data with Large Language Models

2023/12/20 by Bertorello, Pablo M. Rodriguez, Laguerre, Jean Rodmond Junior
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Databases (cs.DB) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2312.13264

Abstract

Data is stored in both structured and unstructured form. Querying both, to power natural language conversations, is a challenge. This paper introduces dIR, Discrete Information Retrieval, providing a unified interface to query both free text and structured knowledge. Specifically, a Large Language Model (LLM) transforms text into expressive representation. After the text is extracted into columnar form, it can then be queried via a text-to-SQL Semantic Parser, with an LLM converting natural language into SQL. Where desired, such conversation may be effected by a multi-step reasoning conversational agent. We validate our approach via a proprietary question/answer data set, concluding that dIR makes a whole new class of queries on free text possible when compared to traditionally fine-tuned dense-embedding-model-based Information Retrieval (IR) and SQL-based Knowledge Bases (KB). For sufficiently complex queries, dIR can succeed where no other method stands a chance.

Related

• SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models
• DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search
• CONVERSER: Few-Shot Conversational Dense Retrieval with Synthetic Data Generation
• LD-SDS: Towards an Expressive Spoken Dialogue System based on Linked-Data
• Redefining Information Retrieval of Structured Database via Large Language Models