2026/05/29 by Minghua He, Lingzhe Zhang, Yuan Liu +2 · 1 voice
Computer Science · #Advanced Data Storage Technologies #Data parallelism #Decoding methods #Error Correcting Code Techniques #Executable #Instruction-level parallelism #Parallel Computing and Optimization Techniques #Parallelism (grammar) #Parameterized complexity #Security token #Task parallelism #cs.PF
paper · pdf · doi:10.48550/arxiv.2605.30851
openalex publication_date 2026/05/29 · arxiv published 2026/05/29 · arxiv updated 2026/05/29 · openalex created_date 2026/06/02 · openalex updated_date 2026/07/28
Parallel decoding improves generation efficiency by processing multiple decode positions within a single decode forward, but reported speedups conflate algorithmic token utilization with the system cost of executing multiple positions. We isolate the system side by introducing Near-Free Parallelism (NFP), the maximum number of positions executable at near-free latency. Analyzing Dense FFNs, MoE FFNs, and Attention against an idle-compute baseline, we find that NFP is shaped not by memory-bound resource slack alone, but also by implementation-induced kernel-granularity slack. Based on these mechanisms, we establish a Near-Free Parallelism principle that predicts the NFP boundary from hardware balance and implementation granularity. Validation on representative Dense and MoE models -- spanning both diffusion and autoregressive decoding -- shows that the principle accurately predicts practical NFP boundaries, revealing that the standard idle-compute intuition can over-predict by up to 23x -- offering a system-side budget for parallelism selection and model-system co-design.