How Diffusion Models Actually Work (Without the Math)

Start with the punchline

A diffusion model is a neural network trained to answer one question, over and over: "Given a noisy image, what would a slightly less noisy version look like?"

If you can answer that question well, you can start with pure noise (random static) and iteratively "denoise" it into a real image. That's the whole trick.

The training side

During training, the model is shown millions of real images. Each one has noise added to it in progressive amounts — a little, a bit more, a lot, and finally pure static. The model learns to predict the noise that was added at each step. If it can predict the noise, it can subtract it.

Under the hood: a U-Net neural network with attention layers, trained to minimize the difference between its noise prediction and the actual noise that was added.

The inference side (what you experience)

When you type "a goat wearing sunglasses":

1. Your text is turned into embeddings by a text encoder (CLIP for SDXL, T5 for Flux).
2. A random noise image is generated (this is your "seed").
3. The U-Net predicts the noise, subtracts a fraction of it. Repeat.
4. After 20–30 steps, no more noise. You have your image.

Why prompt tokens have order

The text embeddings are attended to at every step. Because of how attention works, earlier tokens get more consistent attention across the denoising process. Move a word to the front → the model weighs it more.

Why more steps ≠ better

After ~30 steps, most samplers hit diminishing returns. 50 steps rarely looks better than 30. 100 steps almost never does. Save the compute.

Why SDXL is better than SD 1.5

Two U-Nets stacked (base + refiner), a bigger text encoder, and training on 1024×1024 instead of 512×512. That's most of it.

Why Flux is different

Flux replaces the U-Net with a transformer architecture and uses T5 (a language model) as its text encoder instead of CLIP. Result: it treats prompts more like sentences than tags. That's why long descriptive Flux prompts work better than long descriptive SDXL prompts.