LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs

2025/11/09 by Zifan He, He, Zifan, Shengyu Ye +7
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2511.06174

openalex publication_date 2025/11/09 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

The rapid development of large language models (LLM) has greatly enhanced everyday applications. While many FPGA-based accelerators, with flexibility for fine-grained data control, exhibit superior speed and energy efficiency compared to GPUs, recent GPU-specific optimizations have diminished this advantage. When limited to arithmetic-based computation, FPGAs often underperform GPUs due to their comparatively fewer computational resources. To address this challenge, we exploit a key advantage of FPGAs over GPUs: abundant distributed on-chip memory embedded among computational units. We believe that shifting LLM inference from arithmetic-based to memory-based computations through table lookups can improve the efficiency on FPGAs to compete with GPUs. However, existing methods are inefficient or unable to scale and deploy language models due to algorithm and architecture design limitations. This paper introduces LUT-LLM, the first FPGA accelerator that deploy 1B+ language model with memory-based computation, leveraging vector quantization. We construct a performance model, evaluate multiple quantization schemes, and identify activation-weight vector co-quantization as the most effective approach. To support this scheme, LUT-LLM features (1) bandwidth-aware parallel centroid search to reduce decoding latency, (2) efficient 2D table lookups, and (3) a spatial-temporal hybrid design to reduce data caching for a higher throughput table lookup. We develop a training recipe that converts existing models to support table lookups with high accuracy and prototype LUT-LLM for Qwen 3 1.7B model on the AMD V80 FPGA, reducing arithmetic operations by 4× and achieving a 1.10∼3.29× faster generation speed and a 3.05∼ 6.60× higher energy efficiency than GPUs.

Citations

• A Systematic Characterization of LLM Inference on GPUs
• StreamTensor: Make Tensors Stream in Dataflow Accelerators for LLMs
• Deep Research Agents: A Systematic Examination And Roadmap
• Qwen3 Technical Report
• NeuraLUT-Assemble: Hardware-aware Assembling of Sub-Neural Networks for Efficient LUT Inference
• InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs
• LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator
• CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models
• BitNet a4.8: 4-bit Activations for 1-bit LLMs
• VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models
• Towards Ethical Personal AI Applications: Practical Considerations for AI Assistants with Long-Term Memory
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
• The Llama 3 Herd of Models
• AutoPal: Autonomous Adaptation to Users for Personal AI Companionship
• QTIP: Quantization with Trellises and Incoherence Processing
• A Complete Survey on LLM-based AI Chatbots
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• SpinQuant: LLM quantization with learned rotations
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
• Interactive Continual Learning Architecture for Long-Term Personalization of Home Service Robots
• GPTVQ: The Blessing of Dimensionality for LLM Quantization
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
• FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
• FlashDecoding++: Faster Large Language Model Inference on GPUs
• BitNet: Scaling 1-bit Transformers for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation
• Up or Down? Adaptive Rounding for Post-Training Quantization
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Attention Is All You Need
• SQuAD: 100,000+ Questions for Machine Comprehension of Text

Related

• LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs
• FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
• TerEffic: Highly Efficient Ternary LLM Inference on FPGA
• Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA
• AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design