Liger Kernel: Efficient Triton Kernels for LLM Training

2024/10/14 by Pin-Lun Hsu, Yun Dai, Hsu, Pin-Lun +15 · 37 citations
Computer Science · #Image Processing and 3D Reconstruction #Image and Object Detection Techniques #Handwritten Text Recognition Techniques

paper · pdf · doi:10.48550/arxiv.2410.10989

Abstract

Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.

Cited by

• LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs
• Instruction-Tuning Open-Weight Language Models for BPMN Model Generation
• Motif-2-12.7B-Reasoning: A Practitioner's Guide to RL Training Recipes
• TritonForge: Profiling-Guided Framework for Automated Triton Kernel Optimization
• Automatic Essay Scoring and Feedback Generation in Basque Language Learning
• To Think or Not to Think: The Hidden Cost of Meta-Training with Excessive CoT Examples
• UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
• From FLOPs to Footprints: The Resource Cost of Artificial Intelligence
• Co-Training Vision Language Models for Remote Sensing Multi-task Learning
• MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
• NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
• Gaperon: A Peppered English-French Generative Language Model Suite
• Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
• Arcee Trinity Large Technical Report
• JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training
• DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• Training Large Language Models To Reason In Parallel With Global Forking Tokens
• On The Fragility of Benchmark Contamination Detection in Reasoning Models
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
• Genome-Factory: A Library for Tuning, Deploying, and Interpreting Genomic Foundation Models
• VARCO-VISION-2.0 Technical Report
• AttestLLM: Efficient Attestation Framework for Billion-scale On-device LLMs
• Intern-S1: A Scientific Multimodal Foundation Model
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Faster and Memory-Efficient Training of Sequential Recommendation Models for Large Catalogs
• VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
• MD-LLM-1: A Large Language Model for Molecular Dynamics
• Docopilot: Improving Multimodal Models for Document-Level Understanding

Related

• LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models
• TritonRL: Training LLMs to Think and Code Triton Without Cheating
• KernelBench: Can LLMs Write Efficient GPU Kernels?
• GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length
• CoLLiE: Collaborative Training of Large Language Models in an Efficient Way