2024/11/18 by Zeinab Nezami, Maryam Hafeez, Nezami, Zeinab +5
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Networking and Internet Architecture (cs.NI) #Parallel #Performance (cs.PF) #Scientific Computing and Data Management #and Cluster Computing (cs.DC)
paper · doi:10.48550/arxiv.2411.17712
openalex publication_date 2024/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
6G's AI native vision of embedding advance intelligence in the network while bringing it closer to the user requires a systematic evaluation of Generative AI (GenAI) models on edge devices. Rapidly emerging solutions based on Open RAN (ORAN) and Network-in-a-Box strongly advocate the use of low-cost, off-the-shelf components for simpler and efficient deployment, e.g., in provisioning rural connectivity. In this context, conceptual architecture, hardware testbeds and precise performance quantification of Large Language Models (LLMs) on off-the-shelf edge devices remains largely unexplored. This research investigates computationally demanding LLM inference on a single commodity Raspberry Pi serving as an edge testbed for ORAN. We investigate various LLMs, including small, medium and large models, on a Raspberry Pi 5 Cluster using a lightweight Kubernetes distribution (K3s) with modular prompting implementation. We study its feasibility and limitations by analyzing throughput, latency, accuracy and efficiency. Our findings indicate that CPU-only deployment of lightweight models, such as Yi, Phi, and Llama3, can effectively support edge applications, achieving a generation throughput of 5 to 12 tokens per second with less than 50% CPU and RAM usage. We conclude that GenAI on the edge offers localized inference in remote or bandwidth-constrained environments in 6G networks without reliance on cloud infrastructure.