Vector Search with OpenAI Embeddings: Lucene Is All You Need

2023/08/29 by Jimmy Lin, Ronak Pradeep, Lin, Jimmy +5 · 2 voices · 2 citations
Computer Science · #Advanced Text Analysis Techniques #Time Series Analysis and Forecasting #cs.IR

paper · pdf · doi:10.48550/arxiv.2308.14963

openalex publication_date 2023/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.

Cited by

• Exqutor: Extended Query Optimizer for Vector-augmented Analytical Queries
• LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Models

Discussions

• Vector Search with OpenAI Embeddings: Lucene Is All You Need [hn, 92 points, 63 comments]
• Vector Search with OpenAI Embeddings: Lucene Is All You Need [hn, 5 points, 0 comments]

Related

• Lucene for Approximate Nearest-Neighbors Search on Arbitrary Dense Vectors
• End-to-End Retrieval with Learned Dense and Sparse Representations Using Lucene
• HAKES: Scalable Vector Database for Embedding Search Service
• LEANN: A Low-Storage Vector Index
• VIBE: Vector Index Benchmark for Embeddings