2024/03/07 by Deepti Raghavan, Keshav Santhanam, Raghavan, Deepti +13 · 2 citations
Computer Science · Engineering · Neuroscience · #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #Computation and Language (cs.CL) #Distributed #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Neural Networks and Applications #Parallel #Robotics and Automated Systems #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2403.04311
openalex publication_date 2024/03/07 · openalex created_date 2024/03/09 · openalex updated_date 2026/07/28
Compound AI applications chain together subcomponents such as generative language models, document retrievers, and embedding models. Applying traditional systems optimizations such as parallelism and pipelining in compound AI systems is difficult because each component has different constraints in terms of the granularity and type of data that it ingests. New data is often generated during intermediate computations, and text streams may be split into smaller, independent fragments (such as documents to sentences) which may then be re-aggregated at later parts of the computation. Due to this complexity, existing systems to serve compound AI queries do not fully take advantage of parallelism and pipelining opportunities. We present Alto, a framework that automatically optimizes execution of compound AI queries through streaming and parallelism. Bento introduces a new abstraction called nested ancestry, a metadata hierarchy that allows the system to correctly track partial outputs and aggregate data across the heterogeneous constraints of the components of compound AI applications. This metadata is automatically inferred from the programming model, allowing developers to express complex dataflow patterns without needing to reason manually about the details of routing and aggregation. Implementations of four applications in Alto outperform or match implementations in LangGraph, a popular existing AI programming framework. Alto implementations match or improve latency by between 10-30%.