Idea 1: text becomes tokens
Models do not read letters or whole words. Text is chopped into tokens: common word pieces. "Unbelievable" might become "un" + "believ" + "able"; "mango" is likely one token. A large model knows roughly 50,000 to 200,000 tokens, and everything you have ever typed to a chatbot was converted into a list of token numbers before the model saw it.
Idea 2: meaning becomes numbers (embeddings)
Each token maps to an embedding: a long list of numbers (hundreds or thousands) encoding its meaning. Words used in similar ways get similar numbers, so "sea" sits near "ocean" and far from "homework". These are learned by gradient descent, the Week 5 algorithm, and they are why models can handle words they rarely saw: neighbours in number-space share behaviour.
Idea 3: attention, the superpower
Here is the problem attention solves. In the sentence "The mango that Marcus picked was sweet because it was ripe", what does "it" mean? You know instantly: the mango, not Marcus. A transformer figures that out with attention: every token computes a relevance score against every other token, and pulls in information from the ones that matter most. Layer after layer of this, and the model builds a rich picture of who did what to whom, across a whole conversation.
🔦 Tap a dashed word and watch where its attention goes:
The darker a word glows, the more attention the tapped word pays to it.
Attention with real numbers: a three-word example
Let's shrink attention until you can do it by hand. Take the sentence "the goat escaped" and follow the word "escaped" as it asks: who did the escaping? Give each word a one-number embedding (real models use thousands of numbers per word, we will use one): "the" = 2, "goat" = 6, "escaped" = 9.
- Step 1, score everyone. "escaped" compares itself with every word in the sentence and gets raw relevance scores: "the" scores 1 (grammar glue, rarely matters), "goat" scores 8 (subjects matter a LOT to verbs), and "escaped" itself scores 1.
- Step 2, turn scores into fractions. The scores add up to 1 + 8 + 1 = 10, so squash them into weights that sum to 1: "the" gets 1/10, "goat" gets 8/10, "escaped" gets 1/10. (Real transformers use a function called softmax for this squash; the spirit is identical.)
- Step 3, mix the meanings. Blend the three embeddings using those weights.
Look at that result. 5.9 sits almost on top of "goat" (6). After one attention step, the word "escaped" is CARRYING the goat's meaning inside it, so the model now "knows", in pure arithmetic, who escaped. Stack dozens of these layers and you get a model that can track characters through a whole story. And here is the Week 5 connection: nobody hand-picked that score of 8. The numbers that produce the scores are ordinary network weights, tuned by gradient descent until the scores that help prediction win.
The recipe that turns any list of raw scores into positive fractions that add up to exactly 1, ready to use as attention weights or as next-token probabilities.
Older language models read with a fixed, short memory: only the last handful of words could influence the next prediction. But language does not keep related words close together. In "The mango that Marcus picked from the tree behind the school was sweet", the words "mango" and "sweet" are ten words apart, and a fixed window has forgotten the mango before it reaches "sweet". Attention removes the distance limit entirely: RELEVANCE decides what gets consulted, not position. That one change is a big part of why models went from autocomplete toys to systems that can follow a 50-page document.
The word sounds psychological, so it is easy to picture the model consciously focusing on things, the way you focus in class. It is not doing that. "Attention" is the weighted-average arithmetic you just did by hand: score, squash into fractions, multiply, add. There is no experience of noticing, no inner spotlight, just billions of these little averages per response. Researchers borrowed a human word for a mechanical trick, and the name stuck.
The last step: from scores to a choice (temperature)
After all that thinking, the model outputs a probability for every token in its vocabulary: "next token is 'jump' 55%, 'sing' 25%, …". Then it samples. The temperature setting controls how adventurous the sampling is: temperature 0 always takes the top choice (predictable, sometimes boring), higher temperatures gamble on less likely tokens (creative, sometimes unhinged). When an AI gives you a weird answer, you have seen a low-probability token win the lottery.
The whole journey of one message
Everything in this lesson chains into a single pipeline. When you hit send, this happens, then repeats for every token of the reply:
🎮 Token Beach
How to play
- A sentence is missing its next token. Four voxel signs show candidate tokens.
- Tap the sign (in 3D, or the buttons below) you believe the model rates most likely.
- You score the REAL probability of your pick: choose the 55% token, get 55 points. 8 rounds, 800 possible.
- Above 600 you are officially better at this than temperature 1.0. 😉
In "The hurricane weakened before it reached Portland", how does a transformer work out what "it" refers to?
1) Text is chopped into tokens; each becomes an embedding (a meaning-vector). 2) Attention lets every token consult every other, which is how context and references work. 3) The model outputs probabilities over all tokens, and temperature decides how boldly to sample. You now know what the T, G, and P in "GPT" stand for: Generative, Pre-trained, Transformer.
Next: images from pure noise. 🎨