Esoteric Language Models: A Family of Any-Order Diffusion LLMs

2025/06/02 by Subham Sekhar Sahoo, Sahoo, Subham Sekhar, Zhihan Yang +18 · 1 voice · 15 citations
Arts and Humanities · Business, Management and Accounting · Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Islamic Finance and Banking Studies #Machine Learning (cs.LG) #Media, Religion, Digital Communication #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2506.01928

openalex publication_date 2025/06/02 · arxiv published 2025/06/02 · openalex created_date 2025/10/10 · arxiv updated 2026/05/28 · openalex updated_date 2026/07/28

Abstract

Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us compute the exact likelihood of MDMs for the first time and, crucially, enables us to introduce KV caching for MDMs while preserving parallel generation for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs establish a new state of the art on the speed-quality Pareto frontier for unconditional generation. We provide the code, model checkpoints, and the video tutorial on the project page: https://s-sahoo.com/Eso-LMs.

Cited by

• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• Scaling Behavior of Discrete Diffusion Language Models
• Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
• No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models
• Planned Diffusion
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
• Enabling Approximate Joint Sampling in Diffusion LMs
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• Set Block Decoding is a Language Model Inference Accelerator
• Any-Order Flexible Length Masked Diffusion
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models

Discussions

• Esoteric Language Models [hn, 3 points, 0 comments]

Related

• A Survey on Diffusion Language Models
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• Scaling up Masked Diffusion Models on Text