Bench360: Benchmarking Local LLM Inference from 360 Degrees

2025/11/12 by Linus Stuhlmann, Stuhlmann, Linus, Mauricio Fadel Argerich +3
Computer Science · Decision Sciences · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Performance (cs.PF) #Scientific Computing and Data Management #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2511.16682

openalex publication_date 2025/11/12 · openalex created_date 2025/11/25 · openalex updated_date 2026/08/03

Abstract

Running LLMs locally has become increasingly common, but users face a complex design space across models, quantization levels, inference engines, and serving scenarios. Existing inference benchmarks are fragmented and focus on isolated goals, offering little guidance for practical deployments. We present Bench360, a framework for evaluating local LLM inference across tasks, usage patterns, and system metrics in one place. Bench360 supports custom tasks, integrates multiple inference engines and quantization formats, and reports both task quality and system behavior (latency, throughput, energy, startup time). We demonstrate it on four NLP tasks across three GPUs and four engines, showing how design choices shape efficiency and output quality. Results confirm that tradeoffs are substantial and configuration choices depend on specific workloads and constraints. There is no universal best option, underscoring the need for comprehensive, deployment-oriented benchmarks.

Citations

• LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
• The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization
• SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark
• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators
• Inference Optimization of Foundation Models on AI Accelerators
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• LLM Inference Unveiled: Survey and Roofline Model Insights
• Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• QLoRA: Efficient Finetuning of Quantized LLMs
• NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Measuring Massive Multitask Language Understanding
• MLPerf Inference Benchmark
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Get To The Point: Summarization with Pointer-Generator Networks
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Related

• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
• LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
• UELLM: A Unified and Efficient Approach for LLM Inference Serving