Training: the forward process
Take a real photo. Add a sprinkle of random noise. Add more. And more, over hundreds of steps, until nothing remains but static. Now train a neural network on a simple task: given a noisy image, predict the noise that was added, so it can be subtracted. That is a supervised learning problem, trained with the exact loss + gradient descent machinery from Week 5. The network never memorises photos; it learns what "less noisy" looks like: edges, skin, sky, texture.
A worked example: diffusion on a 2-pixel "image"
Real images have millions of pixels. Let's use TWO, so you can follow every number. Our "photo" is one bright sand pixel and one dark sea pixel, each a brightness from 0 (black) to 255 (white): [200, 40].
Forward (destroy it): add random noise three times.
- Step 1: noise (+20, −10) → the image becomes [220, 30]. Still obviously sand and sea.
- Step 2: noise (−40, +60) → [180, 90]. Getting murky.
- Step 3: noise (+10, +80) → [190, 170]. Two mid-grey pixels. The picture is gone.
Training: show the network [190, 170] and ask it to predict the noise that was just added. If it answers (+10, +80), perfect score. If not, the error flows back through gradient descent and nudges the weights, Week 5 style, over millions of images and noise levels.
Reverse (generate): now run the movie backwards from the ruined [190, 170]. The network predicts the step 3 noise (+10, +80); subtract it to get [180, 90]. It predicts the step 2 noise (−40, +60); subtract to get [220, 30]. It predicts the step 1 noise (+20, −10); subtract and you are back at [200, 40]. Sand and sea, recovered from grey mush. In real generation the starting static never came from any photo, so the network's noise predictions "recover" an image that never existed. That is the whole magic trick.
To decide which specks in a noisy picture are noise, the network must know what SHOULD be there instead. Skies are smooth, so a lone bright dot in a sky is probably noise. Edges are sharp, faces are roughly symmetric, sand has texture, shadows fall away from the light. You cannot subtract noise well without absorbing all of those rules. So the "boring" cleanup task secretly forces the network to learn the deep statistics of the visual world, the same way next-token prediction secretly forces a language model to learn grammar and facts.
Generating: the reverse process
Generation runs the movie backwards. Start from pure random static. Ask the network: "what noise should I remove?" Subtract a little. Ask again. Twenty to fifty steps later, structure emerges from nowhere: a face, a beach, a voxel turtle. It feels like sculpture: the statue was "already in" the marble, and the network chips away noise until it appears.
Think about restoring an old photograph from your grandparents' living room: faded colours, scratches, a water stain across one corner. You can repair it precisely because you know what faces, church shirts, and verandas are supposed to look like; your knowledge fills in what the damage removed. A diffusion model is that skill pushed to the extreme. It "restores" a photo that is 100% damage, pure static, and its learned knowledge of what images look like supplies everything else.
The belief goes: "the AI finds a similar photo somewhere, or stitches together pieces of stored ones." It is tempting because the outputs look so photographic. But the numbers rule it out: the training set is hundreds of terabytes, while the trained model is a few gigabytes, thousands of times too small to contain the photos. What survived training is not pictures but WEIGHTS: compressed knowledge of edges, lighting, anatomy, and style. At generation time nothing is searched and nothing is pasted. There is only the denoising loop you just walked through, steered by your prompt.
Where does the prompt come in?
Left alone, the reverse process produces SOME image. Your prompt steers it. The text ("a sea turtle surfing a wave, voxel art") is encoded into numbers by a language model, and those numbers are fed into the denoiser at every step, biasing every "what noise should I remove?" answer toward turtle-and-wave shapes. The prompt is not a search query. It is a rudder held the whole way down.
Prompt steering, step by step
🎛️ Lab: be the denoiser
Below is pure static. Drag the slider to run denoising steps and watch a beach scene surface from the noise. Notice WHEN you can first tell what it is; that moment is where the composition gets decided, which is why the early steps matter most.
Step 0: pure static. Every generated image ever made started exactly like this.
What is the diffusion network actually trained to do?
Diffusion models learn from millions of images, many made by working artists who never agreed to it. The model does not store their files, but it absolutely absorbed their styles, and courts and parliaments worldwide are still deciding what is fair. Carry that tension into Lesson 5. A genius knows how the tool works AND who paid the hidden costs.
1) Training: add noise to real images, learn to predict it. 2) Generating: start from static and subtract predicted noise, step by step. 3) The prompt steers every denoising step. 4) Early steps decide the composition; late steps polish details.
Next: prompt engineering, the skill that makes all of this obey you. 🛠️