Classifier-free guidance (CFG) has become an essential component of modern diffusion models to enhance both generation quality and alignment with input conditions. However, CFG requires specific training procedures and is limited to conditional generation. To address these limitations, we propose Token Perturbation Guidance (TPG), a novel method that applies perturbation matrices directly to intermediate token representations within the diffusion network. TPG employs a norm-preserving shuffling operation to provide effective and stable guidance signals that improve generation quality without architectural changes. As a result, TPG is training-free and agnostic to input conditions, making it readily applicable to both conditional and unconditional generation. We further analyze the guidance term provided by TPG and show that its effect on sampling more closely resembles CFG compared to existing training-free guidance techniques. Extensive experiments on SDXL and Stable Diffusion 2.1 show that TPG achieves nearly a 2× improvement in FID for unconditional generation over the SDXL baseline, while closely matching CFG in prompt alignment. These results establish TPG as a general, condition-agnostic guidance method that brings CFG-like benefits to a broader class of diffusion models.
Method Overview
TPG constructs guidance by running two forward passes, one through the original network and one through a version with perturbed token representations, and combining their outputs to amplify generation quality at inference time.
At each denoising step, TPG shuffles intermediate token representations using a norm-preserving permutation matrix S, runs two forward passes (positive and negative), and extrapolates the score by amplifying the difference, yielding CFG-like guidance without any conditional training or architecture modifications.
TPG mirrors the structure of classifier-free guidance, but replaces the unconditional branch with a perturbation branch. The positive score s+θ comes from the unperturbed network; the negative score s−θ comes from the shuffled-token pass. The guidance scale γ controls the strength of the effect.
Run the diffusion network normally on the noisy input xt to obtain s+θ, the standard clean score prediction. No changes are made to the network.
A shuffling matrix S is applied to intermediate token representations inside the network. The operation is linear and norm-preserving: it disrupts the spatial structure of tokens while leaving their magnitudes intact.
The same network, now seeing shuffled (incoherent) representations, produces s−θ, a degraded score corresponding to a spatially scrambled latent. This acts as the "unconditional" branch in the guidance formula.
The final estimate extrapolates beyond the positive score, pushing the sample away from the incoherent negative. The guidance scale γ controls the trade-off between diversity and quality, analogous to CFG's guidance weight.
Shuffling destroys the local spatial coherence that diffusion networks rely on to produce structured outputs, while preserving global token statistics. This creates an effective "bad direction" that the model can be guided away from, mimicking the role of the unconditional branch in CFG. Because the operation requires no conditioning signal, TPG extends CFG-like guidance to settings where conditional training is unavailable or impractical.
Conditional Generation
Text-to-image generation on SDXL with TPG, competitive with CFG in prompt alignment
Unconditional Generation
Unconditional image generation on SDXL, nearly 2× FID improvement over the baseline
TPG's guidance term closely mirrors CFG in frequency content and alignment with the ground-truth noise estimate
Frequency & Cosine Similarity Analysis
Cosine similarity of TPG vs. CFG guidance terms across denoising steps (conditional setting)
Face Generation on Stable Diffusion 2.1
TPG applied to SD 2.1 for unconditional face generation
@inproceedings{NEURIPS2025_60d16678,
author = {Rajabi, Javad and Mehraban, Soroush and Sadat, Seyedmorteza and Taati, Babak},
booktitle = {Advances in Neural Information Processing Systems},
pages = {67153--67175},
publisher = {Curran Associates, Inc.},
title = {Token Perturbation Guidance for Diffusion Models},
volume = {38},
year = {2025}
}