Hogwild! Inference: Parallel LLM Generation via Concurrent Attention

2025/04/08 by Gleb Rodionov, Rodionov, Gleb, Roman Garipov +15 · 33 citations
Computer Science · Materials Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.06261

openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have demonstrated the ability to tackle increasingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work is collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such as voting mechanisms or the explicit creation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate. Our approach allows the LLM instances to come up with their own collaboration strategy for the problem at hand, all the while "seeing" each other's memory in the concurrent KV cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's memory. Hogwild! Inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning-capable LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.

Citations

• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
• Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity
• Parallel Scaling Law for Language Models
• Learning Adaptive Parallel Reasoning with Language Models
• THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models
• Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence
• Dynamic Parallel Tree Search for Efficient LLM Reasoning
• Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding
• LIMO: Less is More for Reasoning
• s1: Simple test-time scaling
• GPT-4o System Card
• LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Llama 3 Herd of Models
• Meta Reasoning for Large Language Models
• Mixture-of-Agents Enhances Large Language Model Capabilities
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?
• Self-Discover: Large Language Models Self-Compose Reasoning Structures
• More Agents Is All You Need
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding
• An LLM Compiler for Parallel Function Calling
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Efficient Streaming Language Models with Attention Sinks
• DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
• ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Better Zero-Shot Reasoning with Role-Play Prompting
• Counterfactually Auditable Lifecycle Certification for Autonomous Agents
• Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation
• Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents
• Let's Verify Step by Step
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• LM vs LM: Detecting Factual Errors via Cross Examination
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Fast Inference from Transformers via Speculative Decoding
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Automatic Chain of Thought Prompting in Large Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• Large Language Models are Zero-Shot Reasoners
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Do Transformers Need Deep Long-Range Memory
• Longformer: The Long-Document Transformer
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Attention Is All You Need
• Neural Machine Translation by Jointly Learning to Align and Translate
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Cited by

• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
• The Era of Agentic Organization: Learning to Organize with Language Models
• Planned Diffusion
• The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling
• A Survey on Parallel Reasoning
• Generalized Parallel Scaling with Interdependent Generations
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
• ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
• ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
• Diversity-Enhanced Reasoning for Subjective Questions
• Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
• SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models
• Get Experience from Practice: LLM Agents with Record & Replay
• Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity
• Learning Adaptive Parallel Reasoning with Language Models
• PaDoc: Layout-Grounded Parallel Decoding for Document Parsing
• Self-Steering Language Models
• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

Related

• KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
• Beyond KV Caching: Shared Attention for Efficient LLMs
• Seesaw: High-throughput LLM Inference via Model Re-sharding
• MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
• Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs