Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization

2025/09/25 by Xu, Yuhang, Liu, Shengzhong, Zhang, Dong +3 · 1 citation
#FOS: Computer and information sciences #Operating Systems (cs.OS)

paper · doi:10.48550/arxiv.2509.21301

Abstract

This paper presents Nova, a real-time scheduling framework for serving agentic vision-language models (VLMs) on a single GPU with balanced per-request latency and overall request process throughput. Our design begins by enabling effective pipelining across vision encode, LLM prefill, and LLM decode stages of VLMs, by exploiting their heterogeneous resource demands during execution and incorporating elastic GPU spatial partitioning among stages to maximally utilize the compute and memory resources. Building on this, we introduce a real-time scheduling algorithm that adaptively calibrates resource allocation among stages based on a Pareto-optimal analysis of the latency-throughput trade-off, allowing the system to sustain responsiveness and resource efficiency under dynamic request loads. To further alleviate GPU memory pressure, we design a lightweight weight offloading strategy for vision encoders that preserves inference efficiency with minimized memory overhead. Extensive evaluations on both synthetic and real-world agent workloads demonstrate that Nova consistently outperforms the state-of-the-art baselines, improving the maximum latency by up to 23.3%, while keeping competitive throughput.

Citations

• Pre³: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
• Responsive DNN Adaptation for Video Analytics against Environment Shift via Hierarchical Mobile-Cloud Collaborations
• From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
• KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
• A Survey on Efficient Inference for Large Language Models
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• ScreenAgent: A Vision Language Model-driven Computer Control Agent
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• CogAgent: A Visual Language Model for GUI Agents
• SGLang: Efficient Execution of Structured Language Model Programs
• Audio-Visual LLM for Video Understanding
• Splitwise: Efficient generative LLM inference using phase splitting
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Demand Layering for Real-Time DNN Inference with Minimized Memory Usage
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Root Mean Square Layer Normalization
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• Attention Is All You Need

Cited by

• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling

Related

• LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
• ActionFlow: A Pipelined Action Acceleration for Vision Language Models on Edge
• HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
• Scheduling Techniques of AI Models on Modern Heterogeneous Edge GPU -- A Critical Review