SpecKCast Spectral Kalman Forecasting for Diffusion Acceleration

Javad Rajabi1,2,3,* Hakki Can Karaimer3 David B. Lindell1,2 Babak Taati1,2 Michael S. Brown3

1University of Toronto 2Vector Institute 3AI Center–Toronto, Samsung

*Work done during an internship at Samsung.

Paper arXiv Code · soon

SpecKCast is a training-free method that accelerates diffusion transformers by forecasting their output in the frequency domain with a lightweight Kalman filter. Each frequency band is forecast on its own clock, and the transformer runs only when the forecast becomes too uncertain, achieving high speedup while preserving generation quality.

Faster samplingup to 6×
Negligible memory overhead0.005–0.05 GiB
Training-freeno fine-tuning or extra parameters
Image and video modelsFLUX.1‑dev, Qwen‑Image, Wan2.1‑14B, HunyuanVideo

Abstract

Diffusion Transformers (DiTs) achieve high-fidelity image and video generation, but they suffer from substantial inference costs due to repeated transformer evaluations across many denoising steps. Feature caching reduces this cost by reusing features across denoising steps. Recent forecasting methods extend this idea by extrapolating features to skipped steps instead of reusing them. However, existing forecasting methods extrapolate the features they predict uniformly, even though the spatial-frequency components of these features change most at different stages of denoising. They also overlook that denoising progresses nonuniformly, so equal step intervals can correspond to very different amounts of change. These mismatches amplify forecast errors as more steps are skipped, limiting the achievable acceleration. We propose SpecKCast, a training-free method that forecasts the model output in the frequency domain. SpecKCast applies a 2D discrete cosine transform (DCT) to the output and tracks each coefficient with a lightweight Kalman filter along a band-specific time coordinate derived from the noise schedule. The filter tracks each coefficient’s value and rate of change, and the transformer is evaluated only when the filter’s forecast uncertainty exceeds a threshold. Experiments on FLUX.1-dev, Qwen-Image, Wan2.1-14B, and HunyuanVideo show that SpecKCast achieves a better trade-off between acceleration and fidelity than existing caching and forecasting methods, reaching up to 6× speedup with less than 0.05 GiB of additional memory.

Visual comparisons

Text-to-image

Text-to-video

Method

SpecKCast reduces the number of transformer evaluations during denoising. Unlike feature-caching methods, which approximate an intermediate activation and still run the remaining layers, SpecKCast forecasts the model output itself: the velocity that the sampler uses. It applies a 2D discrete cosine transform (DCT) to this output, tracks each coefficient with a Kalman filter, and evaluates the transformer when the predicted forecast uncertainty exceeds a threshold.

1

Spatial-frequency decomposition

Different spatial frequencies of the output follow different temporal dynamics: low-frequency content settles early, while high-frequency content remains active until the end of sampling. SpecKCast applies an orthonormal 2D DCT to each channel of the output and partitions the frequency plane into non-overlapping radial bands (seven by default). Each coefficient is forecast as its own temporal signal, and its band determines how its dynamics are modeled. On skipped steps, the forecast is transformed back with the inverse DCT and passed directly to the sampler.

2

Band-specific denoising time

Each coefficient is forecast as a function of time, but the step index does not capture the nonuniform progression of denoising. SpecKCast instead uses the log signal-to-noise ratio from the noise schedule. Because each band changes most in a different phase of denoising (early for low frequencies, late for high ones), this coordinate is warped per band using the band's calibrated signal power. The warped coordinate advances more while the band is expected to evolve and less while it is stable; the diffusion process itself is unchanged.

3

Spectral Kalman forecasting

A Kalman filter tracks each coefficient's value and rate of change. At every step, including skipped ones, the value advances by its rate of change, and the rate of change gradually decays; this damping limits overshooting over long skips. On computed steps, the observed coefficient corrects the state through the Kalman update. Coefficients in a band keep their own state but share the time coordinate, the covariance and the process noise; the process noise is calibrated from the band's temporal activity and adapted during sampling from observed forecast errors.

4

Uncertainty-aware scheduling

A fixed schedule cannot respond to changes in forecasting difficulty across generations and denoising phases. SpecKCast instead combines the per-band forecast uncertainties into a scalar risk and evaluates the transformer at the first step where the predicted risk exceeds a threshold; the first three steps and the last step are always computed. The threshold is set once per compute budget on separate calibration prompts. Because the risk adapts to the errors observed in each generation, easier generations use fewer evaluations and harder ones more, so the budget is met on average rather than for every generation.

For models with classifier-free guidance, SpecKCast forecasts the guided velocity that the sampler uses, so a computed step runs both passes and a skipped step runs neither (Appendix E of the paper).

Frequency bands follow different trajectories

PCA trajectories of the 50 velocity predictions of one FLUX.1-dev generation: full velocity and radial DCT bands B0, B3 and B6 Click to enlarge

Figure 2. PCA trajectories of the 50 velocity predictions from a single FLUX.1-dev generation, projected onto the first two principal components; larger gaps between consecutive points indicate larger changes. (a) Sampled entries of the full velocity. (b–d) Radial DCT bands: B0 moves most in the first step, B3 changes over a broader interval, and B6 moves most after step 40. Each panel has its own PCA basis and normalization. Additional views, prompts and Qwen-Image results are in Appendix G of the paper.

Results

PSNR, SSIM and LPIPS measure fidelity to the 50-step reference with the same prompt and seed; ImageReward and CLIP Score are computed from the image and prompt alone. All baselines use their official implementations.

Text-to-image

FLUX.1-dev at 1024×1024 and Qwen-Image at 1328×1328, DrawBench prompts.

SpecKCast has the highest speedup among the acceleration methods in all but one setting, the best PSNR and LPIPS in all settings and the best SSIM in all but one. At the highest acceleration it is 1.1 dB above the strongest baseline in PSNR on FLUX.1-dev (5.54×) and 2.0 dB on Qwen-Image (6.00×), and against Base, which runs at a similar speed, it is 5.5–9.0 dB higher. Its extra memory is at most 0.01 GiB, against 0.73–18.97 GiB for TaylorSeer, HiCache and Spectrum.

Text-to-video

Wan2.1-14B and HunyuanVideo, VBench prompts and protocol.

SpecKCast has the highest or near-highest speedup in every setting, the best SSIM in all settings and the best PSNR and LPIPS in most, and the best VBench Quality on Wan2.1-14B at every budget. Spectrum has slightly lower LPIPS on Wan2.1-14B at 14 and 10 NFE but uses up to 13.12 GiB of extra memory, against 0.05 GiB for SpecKCast.

NFE: transformer evaluations per sample. Base runs the original sampler with fewer steps to match the NFE. Mem. Δ is the peak GPU memory above Base at the same NFE. VBench Quality aggregates dimensions such as temporal consistency, motion smoothness and per-frame quality. Bold is copied from the paper's tables and does not always mark the column best. Measured on NVIDIA H100 GPUs.

Ablation

Components of SpecKCast on FLUX.1-dev at 14 NFE. The highlighted row is the full method.

Bold marks the best and underline the second best, as in Table 3 of the paper. Fixed: a preset schedule whose gaps grow linearly. Adaptive: uncertainty-aware scheduling. Taylor and Chebyshev are the forecasting rules of TaylorSeer and Spectrum, run on SpecKCast's frequency bands.

Under a fixed schedule, the Kalman and Taylor forecasters perform similarly, and Chebyshev is slower and less accurate. Uncertainty-aware scheduling adds 2.2 dB PSNR over the fixed schedule. Band-specific time warping beats the raw log-SNR by 3.3 dB and the step index by 4.7 dB, and removing the frequency decomposition costs 3.1 dB. All variants except Chebyshev run within 0.1 s of each other.

BibTeX

@misc{rajabi2026speckcast,
      title={SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration},
      author={Javad Rajabi and Hakki Can Karaimer and David B. Lindell and Babak Taati and Michael S. Brown},
      year={2026},
      eprint={2610.26837},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.26837},
}