LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs

2024/09/12 by Han Xu, Yutong Li, Xu, Han +3 · 1 voice · 3 citations
Computer Science · #Embedded Systems Design Techniques #Numerical Methods and Algorithms #Video Coding and Compression Technologies #cs.AR

paper · pdf · doi:10.48550/arxiv.2409.11424

openalex publication_date 2024/09/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have demonstrated remarkable abilities in natural language processing. However, their deployment on resource-constrained embedded devices remains difficult due to memory and computational demands. In this paper, we present an FPGA-based accelerator designed to improve LLM inference performance on embedded FPGAs. We employ post-training quantization to reduce model size and optimize for off-chip memory bandwidth. Our design features asynchronous computation and a fully pipelined accelerator for matrix-vector multiplication. Experiments of the TinyLlama 1.1B model on a Xilinx ZCU102 platform show a 14.3-15.8x speedup and a 6.1x power efficiency improvement over running exclusively on ZCU102 processing system (PS).

Cited by

• PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfiguration
• Implementation and Evaluation of Stable Diffusion on a General-Purpose CGLA Accelerator
• TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs

Discussions

• 인텔이 알테라를 인수했을 때, AMD가 자일링스를 인수했을 때 기대했던 것은 이런 것이었는데: 바로 소프트웨어로 변경가능한 하드웨어 프로그래밍. 하지만 경영의 사정으로 칩을 통합하고 툴을 일반재로 만드는 것은 고수익을 포기하는 짓이 되어서... arxiv.org/abs/2409.11424 [bsky, 0 points, 0 comments]

Related

• Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA
• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
• Designing Efficient LLM Accelerators for Edge Devices
• FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
• Hummingbird: A Smaller and Faster Large Language Model Accelerator on Embedded FPGA