Separating Intelligence from Inference: A Standard for Edge-Native AI Computing

2026/05/13 by Venkat Vinjam, Krishnaiah Narukulla
Computer Science · #cs.DC

paper · pdf

23 pages, preprint, Separating Intelligence from Inference

arxiv created 2026/05/13 · arxiv updated 2026/08/05

Abstract

The artificial intelligence industry has constructed a USD 300 billion centralized data center infrastructure to serve a workload, large language model inference, that does not architecturally require centralization. This paper articulates the central architectural inefficiency of contemporary AI infrastructure: the conflation of model training (irreducibly centralized, capital-intensive, one-time per model version) with model inference (parallelizable, latency-sensitive, recurring per query) on the same physical hardware. We propose the separation principle: intelligence is trained centrally and shipped as software; inference executes on hardware near the data source, at the edge. We quantify the energy implications at civilizational scale and show that a fully edge-resident inference architecture for one billion daily users saves approximately 19 TWh per year and 7.3 megatons of CO2 annually relative to current centralized practice. We specify two new device classes, the Personal AI Computer (PAC) and the Corporate AI Workstation (CAW), with concrete hardware tiers, memory bandwidth requirements, thermal envelopes, and software interfaces. We then describe a reference architectural stack of eight components addressing weight distribution, sovereignty-aware routing, thermal-adaptive quantization, multi-tenant resource management, federated network inference, cryptographic provenance, privacy-preserving telemetry, and distributed context window extension. Several components are the subject of pending United States patent applications by the first author and are presented here as candidate open architectural principles

Citations

• The Llama 3 Herd of Models
• Apple Intelligence Foundation Language Models
• RouteLLM: Learning to Route LLMs with Preference Data
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• LLaMA: Open and Efficient Foundation Language Models
• Fast Inference from Transformers via Speculative Decoding
• Efficiently Scaling Transformer Inference
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
• MLPerf Inference Benchmark
• Energy and Policy Considerations for Deep Learning in NLP
• Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
• Attention Is All You Need
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Differential Privacy
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration