Revisiting the 16× Pretraining Compute Gap between Diffusion and Autoregressive Language Models

Guanghan Wang1, Bing He2, Zhan Shi2, Benoit Dumoulin2, Hanqing Lu2,*, Volodymyr Kuleshov1,*
1Cornell     2Amazon     *Equal supervision
arXiv Code

TL;DR: The widely cited 16× compute gap between diffusion and autoregressive language models largely reflects an outdated scaling-law protocol. Under current practice, the gap falls to between 3× and 8×.

Compute multiple of diffusion language models over autoregressive models as the protocol is revised

Figure 1. Left: the compute multiple c = CDLM/CARM measures how many times more training compute a diffusion language model (DLM) needs than an autoregressive model (ARM) to reach the same compute-optimal negative log-likelihood (NLL). Right: c as the evaluation (ELBO → DUEL), the model design (MDM → BD3LM → E2D2) and the block size are revised.

Our contributions

  • We revisit the scaling laws of diffusion language models (DLMs) under a protocol that reflects current practice: exact likelihood evaluation, block diffusion, an encoder–decoder architecture, and a sweep over the block sizes used in practice.
  • We show that pretraining DLMs is far cheaper than previously assumed. Exact likelihood evaluation reduces the compute multiple to 12.8×; block diffusion with an encoder–decoder architecture reduces it to 7.8× at block size 32; and smaller blocks narrow it to 4.6× at block size 8 and 3.0× at block size 4.
  • We find a simple empirical rule for the optimal decoder depth of encoder–decoder block diffusion, which depends on the block size alone.
  • We show that the gap between variants persists at a held-out budget three times larger than our largest fitted one, and that likelihood improvements carry over to downstream benchmark accuracy.

Where the 16× came from

The 16× figure comes from the IsoFLOP study of SMDM (Nie et al., 2025), which trained a full-sequence, decoder-only masked diffusion model (MDM), scored it with its ELBO, and compared it with an ARM at matched likelihood. Since then, diffusion language models have changed in three ways that this protocol does not reflect:

  • Evaluation. The ELBO is only an upper bound on the NLL, and it scores the model under uniformly random token orders it never uses at inference. DUEL (Turok et al., 2026) instead gives the exact NLL under the sampler the model is deployed with, such as confidence-based unmasking.
  • Block diffusion. Nearly all large-scale DLMs today generate block by block (BD3LM; Arriola et al., 2025), with block sizes between 4 and 32.
  • Architecture. Encoder–decoder block diffusion (E2D2; Arriola et al., 2026) separates the processing of clean and noised tokens, and is in principle cheaper to train than the decoder-only network inherited from ARMs.

We adopt all three, sweep the block sizes b ∈ {4, 8, 16, 32}, count training FLOPs exactly (including the output head and attention), and read the compute multiple within the measured range of the scaling laws rather than by extrapolation. This gives ten model variants and about 1,700 trained models across five budgets from 6×1018 to 1020 FLOPs.

From 16× to 3×, one step at a time

We now show in more detail how each step of the revised protocol improves the compute multiple, i.e., the training efficiency of DLMs relative to ARMs:

  • ELBO baseline: 17.8×. We roughly reproduce the SMDM 16× baseline as 17.8×.
  • Exact likelihood: 12.8×. Scoring the very same checkpoints with DUEL pushes down the compute multiple to 12.8×.
  • Block diffusion: 9.6×. At b = 32, block factorization training helps reduce the compute multiple to 9.6×.
  • Encoder–decoder: 7.8×. At b = 32, E2D2 beats the decoder-only BD3LM in terms of training efficiency, reaching a compute multiple of 7.8×.
  • Smaller blocks: 6.3×, 4.6× and 3.0× at b = 16, 8 and 4.
Fitted compute-optimal NLL scaling laws of the model variants

Figure 2. Compute-optimal NLL of different model variants, solid inside the fitted range and dashed outside.

How deep should the E2D2 decoder be?

E2D2 has one design choice the other variants do not: how to split the layers between the encoder and the decoder. We swept the decoder depth at every grid point, and found that the optimal decoder fraction depends on the block size alone and is decoder-heavy:

L*dec / L  ≈  0.556 + 0.043 log2 b

In practice, one multiplies the rule by the total depth L and rounds to a whole number of decoder layers. At 132 of the 142 grid points with b ∈ {4, 8, 16, 32}, the resulting configuration lies within 10 m-nats of the best DUEL found by the sweep, including all 20 compute-optimal ones.

Does it hold up?

Beyond the fitting range. We trained the ARM and E2D2 (b ∈ {16, 32}) at a held-out budget of 3×1020 FLOPs, three times the largest fitted budget. The gap between E2D2 and the ARM predicted by the fitted laws is reproduced almost exactly.

Downstream benchmarks. On eight zero-shot benchmarks (HellaSwag, ARC-easy, PIQA, SciQ, RACE, SIQA, OpenBookQA and LAMBADA), the order of the variants in mean accuracy follows their order in likelihood.

Held-out budget at 3e20 FLOPs

Figure 3. Laws fitted on budgets up to 1020 FLOPs (dots); stars are the measured values at 3×1020.

Mean benchmark accuracy against compute

Figure 4. Mean accuracy over eight benchmarks against compute for the compute-optimal model of each variant.

Caveats

Our experiments run up to 3×1020 FLOPs with a single seed, and all variants share one learning-rate and batch-size configuration, inherited from SMDM for comparability. DUEL is measured under the top-1 confidence sampler only. See the paper for more details.

BibTeX

@article{wang2026revisiting,
  title   = {Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models},
  author  = {Wang, Guanghan and He, Bing and Shi, Zhan and Dumoulin, Benoit and Lu, Hanqing and Kuleshov, Volodymyr},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}