omni-macos: On-Device Omni-Modal Search on Apple Silicon

2026/08/06 by Han Xiao
Computer Science · #cs.IR

paper · pdf

16 pages, 5 figures, 8 tables

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

A search engine that embeds text, code, documents, images, audio and video into the same representation space has to run its encoder and keep its index somewhere, and almost every component built for the purpose assumes a server. We present omni-macos, which runs that whole engine, encoder, index and store, on the Mac the files are already on, so no file, query or vector ever leaves the machine. It keeps a background indexer and an interactive search box inside one memory budget the user sets: it re-encodes only the chunks an edit changes, hands the GPU smaller units while the user is typing, answers queries from a quantized replica with exact rescoring, and propagates that budget to the allocators that draw on unified memory. We measure every mechanism on five Macs spanning an eightfold range of accelerator width and a thirty-twofold range of memory, each one indexing its own local files.

Citations

• Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
• Quantization for Vector Search under Streaming Updates
• Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS
• Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
• Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantization
• A Faster Generalized Two-Stage Approximate Top-K
• 4bit-Quantization in Vector-Embedding for RAG
• Locally-Adaptive Quantization for Streaming Vector Search
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory
• LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• ImageBind: One Embedding Space To Bind Them All
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Billion-scale similarity search with GPUs
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs