2023/07/10 by Zhihe Zhao, Neiwen Ling, Zhao, Zhihe +5 · 1 citation
Computer Science · Neuroscience · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2307.04339
openalex publication_date 2023/07/10 · openalex created_date 2023/07/12 · openalex updated_date 2026/07/28
Many applications such as autonomous driving and augmented reality, require the concurrent running of multiple deep neural networks (DNN) that poses different levels of real-time performance requirements. However, coordinating multiple DNN tasks with varying levels of criticality on edge GPUs remains an area of limited study. Unlike server-level GPUs, edge GPUs are resource-limited and lack hardware-level resource management mechanisms for avoiding resource contention. Therefore, we propose Miriam, a contention-aware task coordination framework for multi-DNN inference on edge GPU. Miriam consolidates two main components, an elastic-kernel generator, and a runtime dynamic kernel coordinator, to support mixed critical DNN inference. To evaluate Miriam, we build a new DNN inference benchmark based on CUDA with diverse representative DNN workloads. Experiments on two edge GPU platforms show that Miriam can increase system throughput by 92% while only incurring less than 10% latency overhead for critical tasks, compared to state of art baselines.