PipeANN-Filter: An Efficient Filtered Vector Search System on SSD

2026/05/18 by Hao Guo, Jiwu Shu, Youyou Lu · 1 voice
Computer Science · #Advanced Data Storage Technologies #Caching and Content Delivery #Data structure #Latency (audio) #Leverage (statistics) #Network Packet Processing and Optimization #Probabilistic logic #Reduction (mathematics) #Search algorithm #cs.DB #cs.OS

paper · pdf · open access · doi:10.48550/arxiv.2605.17992

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/05/18 · arxiv published 2026/05/18 · arxiv updated 2026/05/18 · openalex created_date 2026/05/20 · openalex updated_date 2026/07/28

Abstract

We propose PipeANN-Filter, an efficient filtered vector search system on SSD. Unlike existing systems that explore only valid vectors (i.e., those satisfying the attribute constraints) during search, PipeANN-Filter explores a superset of valid vectors, and performs attribute verification after getting the top-k closest result vectors. This allows PipeANN-Filter to leverage probabilistic data structures (e.g., Bloom filters) to identify the superset, trading off a small number of false-positive vector explorations for a massive reduction in SSD I/O for attribute reading. Evaluations show that PipeANN-Filter improves search latency and throughput compared to state-of-the-art systems. PipeANN-Filter is open-source at https://github.com/thustorage/PipeANN

Citations

• Scalable Disk-Based Approximate Nearest Neighbor Search with Page-Aligned Graph
• UNIFY: Unified Index for Range Filtered Approximate Nearest Neighbors Search
• Results of the Big ANN: NeurIPS'23 competition
• ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data
• SPFresh: Incremental In-Place Update for Billion-Scale Vector Search
• Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Search
• SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Data
• The Design and Implementation of a Real Time Visual Search System on JD E-commerce Platform
• Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors
• Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graph
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• Cuckoo Filter
• Efficient Estimation of Word Representations in Vector Space
• Space/time trade-offs in hash coding with allowable errors

Discussions

• Hao Guo, Jiwu Shu, Youyou Lu: PipeANN-Filter: An Efficient Filtered Vector Search System on SSD https://arxiv.org/abs/2605.17992 https://arxiv.org/pdf/2605.17992 https://arxiv.org/html/2605.17992 [bsky, 0 points, 0 comments]

Related

• GateANN: I/O-Efficient Filtered Vector Search on SSDs
• Zoom: SSD-based Vector Search for Optimizing Accuracy, Latency and Memory
• MicroNN: An On-device Disk-resident Updatable Vector Database
• Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN
• Vector-Centric Machine Learning Systems: A Cross-Stack Approach