Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

2025/04/04 by Erik Johannes Husom, Husom, Erik Johannes, Arda Göknil +13 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #Data Stream Mining Techniques #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2504.03360

openalex publication_date 2025/04/04 · openalex created_date 2025/10/15 · openalex updated_date 2026/07/28

Abstract

Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.

Cited by

• Smart but Costly? Benchmarking LLMs on Functional Accuracy and Energy Efficiency
• Metrics and evaluations for computational and sustainable AI efficiency
• Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizons
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• Agentic AI Reasoning for Mobile Edge General Intelligence: Fundamentals, Approaches, and Directions
• AQUA-LLM: Evaluating Accuracy, Quantization, and Adversarial Robustness Trade-offs in LLMs for Cybersecurity Question Answering
• Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

Related

• An Evaluation of LLMs Inference on Popular Single-board Computers
• LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi
• Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
• Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge
• From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference