The Stretto Execution Engine for LLM-Augmented Data Systems

2026/02/04 by Gabriele Sanmartino, Matthias Urban, Paolo Papotti +1 · 1 voice
Computer Science · #cs.DB

paper · pdf

arxiv published 2026/02/04 · arxiv updated 2026/02/06

Abstract

LLM-augmented data systems enable semantic querying over structured and unstructured data, but executing queries with LLM-powered operators introduces a fundamental runtime-accuracy trade-off. In this paper, we present Stretto, a new execution engine that provides end-to-end query guarantees while efficiently navigating this trade-off in a holistic manner. For this, Stretto formulates query planning as a constrained optimization problem and uses a gradient-based optimizer to jointly select operator implementations and allocate error budgets across pipelines. Moreover, to enable fine-grained execution choices, Stretto introduces a novel idea on how KV-caching can be used to realize a spectrum of different physical operators that transform a sparse design space into a dense continuum of runtime-accuracy trade-offs. Experiments show that Stretto outperforms state-of-the-art systems while consistently meeting quality guarantees.

Citations

• Featurized-Decomposition Join: Low-Cost Semantic Joins with Guarantees
• Implementing Semantic Join Operators Efficiently
• Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
• The Llama 3 Herd of Models
• Finch: Prompt-guided Key-Value Cache Compression
• Semantic Operators: A Declarative Model for Rich, AI-based Data Processing
• RouteLLM: Learning to Route LLMs with Preference Data
• SnapKV: LLM Knows What You are Looking for Before Generation
• Optimizing LLM Queries in Relational Data Analytics Workloads
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• CAESURA: Language Models as Multi-Modal Query Planners
• DiffML: End-to-end Differentiable ML Pipelines
• DARTS: Differentiable Architecture Search
• Adam: A Method for Stochastic Optimization
• THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL

Discussions

• 🚀 New: The Stretto Execution Engine for LLM-Augmented Data Systems. LLM operators create a runtime ↔ accuracy trade-off in query execution. We address it with a novel optimizer, for end-to-end qualit [bsky, 4 points, 1 comments]

Related

• Kalypso: Relational LLM Serving
• Accelerating LLM Inference with Precomputed Query Storage
• Optimizing LLM Queries in Relational Data Analytics Workloads
• Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
• The Unreasonable Effectiveness of LLMs for Query Optimization