Model Field Notes

The foundations have actually converged

It may be surprising, but today’s leading image and video models are Transformers inside too. Cut a picture into small squares (patches), treat them as a sequence, and the same machine that handles text will handle it. So a model that draws and a model that writes are, as components, remarkably alike.

What differs is what that machine was made to practise.

Three dials

That last column is, near enough, the rate of progress. Where a machine can mark the work, practice runs millions of times without waiting for a person. That is essentially why code and mathematics have raced ahead.

A language model writes left to right, one word at a time. An image model works on the whole frame at once, from blur towards clarity. The approach is called a diffusion model.

Training is guessing the noise that was added

What it does fits in a few lines.

The correct answer — the noise that was added — is something the process created itself, so no human has to write it. It is the same shape as “the answer is inside the data” from No.01. Images, too, could therefore be scaled without annotation.

Why words can steer it

The key is putting words and pictures in the same coordinate space. Collect vast numbers of image-caption pairs and train the coordinates so that matching pairs sit close together and unrelated ones far apart (contrastive learning — CLIP is the well-known example).

Once that space exists, a phrase like “a harbour at dusk” becomes a vector, and that vector serves as the landmark the denoising steers towards. Language becomes the compass for carving out the picture.

Latent space — pixels are too heavy

Working directly on 1024×1024 pixels does not pay. So another network first compresses the image to roughly an eighth of its width and height — a latent space — diffusion runs there, and the result is expanded back into an image at the end. That trick is why image generation runs on a personal machine at all.

Why text and hands used to go wrong

Diffusion accumulates local plausibility. If a patch looks right, the loss goes down. But written text demands an exact sequence — one wrong stroke and the meaning breaks. Hands are the same: counting to five is not something local plausibility can enforce.

The recent improvement came from enlarging the text encoder, improving caption quality and growing the model. Less a change of mechanism than resources poured into the known weak spot.

Video is not “a row of images”. Adding one more dimension changes the nature of the difficulty.

Difficulty 1: volume

512×512 at 24 frames a second for five seconds is more than a hundred times the information in a single picture. The quadratic cost from No.01, chapter II applies directly, so doubling the length costs far more than double. That wall is why generated clips tend to stop at seconds rather than minutes.

Difficulty 2: consistency

Draw each frame independently and the person changes between one moment and the next. So attention has to run along the time axis too: a patch at one instant is decided while looking at the same region in neighbouring frames.

In practice the dominant design now cuts space and time into patches and hands them to a Transformer (DiT). That means the empirical rule proven on language models — bigger gets better — carries straight over to video.

Physics was never taught

A common misunderstanding: there is no equation for gravity or fluid flow inside the model. From enormous amounts of footage it has learned what falling looks like and how water tends to move — regularities of appearance, nothing more.

Which is why familiar scenes look startlingly natural while objects vanish, multiply, or swap depth. Plausibility is satisfied; conservation is not. The root is the same as hallucination in No.01, chapter IV.

1. Execution is the answer key

Prose can only be judged by people; code is judged by whether the tests pass. So the reward model (No.01, chapter III) can be skipped and correctness itself used as the reward. Practice that never waits for a human converts directly into speed of improvement.

2. Practising the gap, not just the continuation

Editor completion needs more than writing onward; it has to look before and after the cursor and fill the middle. So training deliberately hides a span and asks the model to restore it from both sides — fill-in-the-middle.

3. Long context, and what lies outside it

In real projects the definition of a function may be hundreds of lines away, or in another file. Extending the context window is not enough on its own; it only becomes practical alongside machinery that finds the relevant files and hands them over. That is exactly the job of the agent in No.01, chapter VI.

4. Strict form is an advantage

One missing bracket stops everything — which is another way of saying mistakes are machine-detectable. “This prose feels off” cannot be measured; “this does not run” can. Domains that can be measured accumulate improvement.

Taking apart the part that looks like understanding intent. The short answer: it is less understanding than conditioning.

1. Pre-training read an enormous amount of conversational form

The web is full of questions and answers, problems and explanations, requests and advice. In the course of predicting the next word, the shape of “asked like this, answered like that” is itself absorbed.

2. Post-training puts it on the answering side

A raw model may respond to a question with more questions. Training on instruction and answer pairs, then polishing on human judgement (No.01, chapter III), is what fixes the stance of being asked and answering.

3. The actual inference — weighting the clues

An input carries far more than what it literally says.

• Vocabulary and register — who is asking, and how close the tone should be
• The language and framework of any pasted code — the environment being assumed
• Presence or absence of jargon — how much to unpack
• The preceding exchange — what is being built right now

Attention weights those clues and produces the most likely continuation in this context. Reading intent, concretely, is that weighting.

4. Noticing ambiguity

Recent models have also learned to ask back when a premise is missing. But that is less “knowing what it does not know” than the learned fact that asking back scored better. Do not lean on it; writing the premise yourself is more reliable.

Finding bugs works because three layers of a different nature stack up.

Layer 1: memory of the classic shapes

Training data contains a great many fix commits from public repositories — the broken state and the repaired state, side by side. From these come the classic shapes: the boundary off by one, the unchecked null, the resource never released, the swallowed exception.

This is detection as knowledge: fast, and broad in coverage.

Layer 2: reading for mismatch of meaning

Closer to human review. It compares names, comments, types and what the caller expects against what the code actually does.

A linter can only report that floor is a valid function. Catching this requires holding the comment and the code in the same context and checking one against the other.

Layer 3: actually running it

Reading has a ceiling. Writing a test and running it, reproducing the error, reading a stack trace to narrow the hypothesis — this is the layer the agent from No.01, chapter VI handles. Only with this step, which turns a guess into a check, does the accuracy become usable.

Strong and weak spots

Dividing the work with static analysis

Linters and type checkers are a set of rules: exhaustive and reproducible, but blind outside the rules. A model catches the inconsistency you would notice on reading, but without exhaustiveness, and it may say something different on the next run.

The properties are opposite, so the realistic move is not replacement but layering: leave the mechanically measurable to static analysis, and ask the model about the gap between intent and implementation.