The machine that learns to answer

Imagine a friend who has read every book in the library — and has one strange habit: whatever you say, they just keep the sentence going. You ask "what colour is the fox?" and they cheerfully reply "…what colour is the swan? where does the mole live?" — because in the books they read, questions come in lists, and one question is usually followed by another one.

They're not being rude. Continuing is the only game they have ever played. That friend is every freshly trained language model on Earth.

Part 1 of this series showed how the giants learn: hide a piece of the text, guess it, check, repeat — a trillion times. That game builds a model that can continue any text with uncanny fluency. But notice what the game never contains: nobody ever asks the model for anything. There is no "you", no request, no reply. Text goes in, likely-next-text comes out.

So a raw pretrained model — a base model — treats your question as just more text to extend. Ask early GPT models a question and they would often reply with a list of similar questions: the internet is full of FAQs and quizzes, and a question is, statistically, excellent evidence that more questions are coming.

The fix turns out to be almost embarrassingly small compared to pretraining, and that smallness is the deepest fact on this page.

The pipeline every modern assistant goes through: pretrain → supervised fine-tuning → preference tuning. Pretraining (Part 1) spends nearly all the compute and learns nearly all the capability — next-token prediction over web-scale text. SFT, this page, is stage two: continue training the same weights with the same loss on a tiny, curated set of demonstrations — conversations written the way you wish the model would write. Stage three (the companion page on RL) shapes judgment with feedback instead of examples.

Keep the proportions in mind: InstructGPT pretrained on hundreds of billions of tokens, then ran SFT on about thirteen thousand demonstrations. That ratio — a sliver of the data changing nearly all of the behavior — is why researchers argue SFT mostly redirects capability rather than creating it.

Formally, a base model is an estimate of the pretraining distribution p(next token | context); helpfulness is a property of a different conditional — the distribution of assistant replies given requests — which is underrepresented and entangled in web text. SFT is a distribution shift accomplished by continued maximum-likelihood training on a small sample from the target conditional (Ouyang et al., 2022, InstructGPT — following the instruction-tuning line of FLAN and T0, Wei et al. 2021, Sanh et al. 2021). The striking empirical fact is the sample efficiency: LIMA (Zhou et al., 2023) reports strong instruction-following from just 1,000 curated demonstrations, and frames the superficial alignment hypothesis — a base model already possesses the capabilities, and alignment teaches mostly the format and style of the interface. This page's toy model will let us dissect that claim with unusual cruelty, because at our scale the "capability" and the "format" come apart and can be measured separately.

How do you teach the habit of answering? The same way you teach handwriting: with a copybook. You write out a question with its answer, in exactly the shape you want — and the reader practices tracing it, over and over, until the shape becomes a habit.

A demonstration is a question paired with the answer you wish the model would give, written in a fixed costume: q: before the question, a: before the answer. Then — and this is the part people never believe — you train on it with the exact same next-character game as pretraining. No new loss, no new machinery. Same pencil, different copybook.

Two engineering details carry most of the weight. First, the template: the q: / a: markers are this page's miniature chat template. Real assistants use special tokens for turns and an end-of-turn marker — which is also how a model learns to stop talking; our equivalent is the newline that ends each answer. Second, loss masking: compute the loss only on the answer characters, not the prompt. The model should learn to produce answers, not to imitate questions — we'll put this dial on a test bench in section 5.

SFT objective, in full: maximum likelihood on response tokens conditioned on the prompt — ∑ log p_θ(y_t | x, y_<t), i.e. behavior cloning of the demonstrator policy. Everything else is data curation. The known structural limits are inherited from imitation learning: quality is capped by the demonstrator, and training is teacher-forced, so the model never practices recovering from its own mistakes (exposure bias; DAgger is the classic analysis). Those limits are precisely what stage three exists to lift.

Real-world scale check: what you just ran is the entire recipe of 2022-era assistant training, miniaturized. InstructGPT's stage two was ~13,000 demonstrations written by hired labelers; Alpaca did it with 52,000 demonstrations generated by a bigger model — which you may recognize as Part 2 of this series wearing a chat template: fine-tuning a student on a teacher's outputs is distillation by another name. And LIMA pushed the other direction: 1,000 immaculate demonstrations, nothing else, and a remarkably capable assistant.

The reader aced its copybook. But here's the detective question: did it learn how to answer questions… or did it just memorize those sixteen answers? There's a way to find out: ask it questions that were not in the copybook — about animals it knows perfectly well from its storybook.

Time to be scientists about the thing we just made. It answers its 16 practiced questions flawlessly. Two very different explanations fit that fact: it learned the skill of answering, or it learned sixteen strings. The experiment that separates them: quiz it on the other 74 questions — same storybook animals, same three question shapes, facts it recited perfectly before the fine-tune. Place your bet in the panel before you peek.

The probe set factorizes cleanly: (entity, relation) pairs it practiced, versus pairs it never saw in SFT — including unseen relations of practiced entities. If SFT were unlocking pretrained knowledge, un-practiced pairs should score far above chance; if SFT is installing a format, we should see answer-shaped output whose content is noise. At our scale there is also a mechanistic prediction. Answering requires copying the entity name from the question into the answer — and a fixed-window MLP has no attention, no mechanism for content-based copying. Watch for exactly that failure.

So the copybook taught it the shape of answering — it now says "a: the … is …" like a champion — but the shape comes out filled with the wrong animal's facts. And something sadder happened too: while it practiced answering, it started forgetting the storybook. Teaching the habit wasn't free.

Hold on to both halves of this result, because together they are the honest heart of the whole page. The habit generalized — most fresh questions now get an answer-shaped reply. The knowledge didn't travel — the content of those replies is chance. And the model paid tuition: roughly half of the 90 storybook facts it could recite before the fine-tune now come out wrong. Sixteen answers didn't just add a habit; they overwrote pieces of a library.

Now the crucial scale caveat, stated carefully. In real LLMs, the LIMA result shows held-out generalization working: a thousand demonstrations, and the model answers questions on topics none of them covered. The difference is machinery: a transformer has attention — a content-addressed copying and retrieval mechanism that pretraining has already built — so SFT only has to teach when to invoke it. Our MLP has no such mechanism, so the same procedure teaches format and nothing else. That makes the toy a clean dissection, not a scale model: it shows what SFT alone contributes (the habit) when there is no retrieval machinery for it to recruit. The forgetting half, though, scales fine: it is ordinary catastrophic forgetting, the reason real labs mix pretraining data into SFT and call the behavior drop from alignment an alignment tax.

Measured, for the record (10 seeds): un-practiced pairs 0 exact in every configuration tested — including 8× the compute and 4× the data — with attribute-level accuracy at or below the uniform-chance floor; answer-shaped output on 60.8% of fresh questions (the a: marker); bare-prompt recall on un-practiced entities falling from 100% to 21–44%. The entity-copy failure is directly visible: the fine-tuned model echoes the question's entity into its answer at a rate near zero — it answers with practiced entities instead. This is the superficial-alignment hypothesis run in a regime where it becomes exact: alignment taught the surface, because the surface was all this architecture could share between examples. The open question at real scale — how much of instruction-following is elicitation versus new capability — is the same question the RL companion page meets again as pass@1 vs pass@k.

One more experiment, and it's the spooky one. We'll ask about an animal that doesn't exist — not in the storybook, not in the copybook, not anywhere. A puma. The reader has never seen that word in its life.

Think about what the copybook actually taught: every question gets an answer. All sixteen demonstrations answer confidently; not one says "I don't know" — so "I don't know" is not in the habit. What happens when the habit meets a question with nothing behind it? Bet first.

This is hallucination's origin story in miniature. SFT is maximum-likelihood on demonstrations; if the demonstration distribution contains only confident answers, the fine-tuned conditional puts its mass on confident answers — regardless of the model's epistemic state, which the loss never sees. A base model, asked about the unknown, just rambles on-distribution; it is the fine-tune that installs the reflex of answering. Refusal, calibration, "I don't know" — each is a behavior that must itself be demonstrated or reinforced; none appears for free.

It answers. It always answers. Sometimes the answer is scrambled, sometimes it's a perfectly confident sentence about the wrong animal — but it never once says "I've never heard of a puma." Saying "I don't know" is its own habit, and nobody put it in the copybook.

Notice the exact shape of the failure, because it's subtler than "it makes things up." Sometimes — about one fake probe in thirteen, across many runs — it reaches for a similarly-spelled real animal and hands you that animal’s true fact with full confidence (in one of our measured runs, asking about the wisp returned the wasp’s real colour). And be fair to it: this isn't lying — the training signal never contained a difference between "answer you know" and "answer you don't." Fixing that — teaching honesty as a behavior — is feedback's job, not the copybook's: that story continues in the RL companion page.

Measured (10 seeds): fake-entity questions drew an answer-shaped response 0% of the time before SFT and 67.5% after; a fully fluent made-up answer about a quarter of the time; a refusal or "unknown" response, 0% — structurally impossible, since no such string was ever demonstrated. A caution for anyone building the same demo: 7.5% of fake probes elicited a true fact about a similarly-spelled real animal (one measured run answered the wisp question with the wasp's true colour) — nearest-neighbour retrieval, not invention; captions here are generated from the measured output, not from what the story would prefer. The real-scale echo is well documented: RLHF-era models over-answer because demonstrations and preference data under-represent abstention, and "hallucination as a training artifact" (Schulman's framing) remains an active repair area — via refusal demonstrations, honesty rewards, and calibration probes.

Two last experiments for the tinkerers. First: when the reader practices the copybook, should it trace the questions too, or only the answers? Second: if a little practice is good, is a lot of practice better?

Every SFT recipe has two dials that look boring and aren't. Dial one — what the loss touches. Train on the whole demonstration and half your gradient is spent learning to write questions; mask the prompt and every step goes into answering. Dial two — how hard you press. The learning rate and step count decide how much of the old library survives the new habit. Both dials run live below; every number is computed at press time.

The surprise most people lose the bet on: time barely matters; pressure is everything. Training ten times longer leaves the library almost exactly where the normal run left it — the forgetting happened in the first few dozen steps, before the copybook was even learned. But turn the learning rate up to 0.05 and one ordinary-length run wipes out almost ninety percent of the library while still acing the copybook. What long practice does slowly grow is parroting: fresh questions increasingly get answered with a memorized copybook line, word for word.

Practitioner's translation: SFT learning rates run tiny for a reason — the damage dial is lr, and the tax is front-loaded, so "just train less" is not a mitigation. The classic mitigation — mixing replayed pretraining data into the fine-tune — is real at scale, but don't assume it's free: in this toy's budget, replay measurably cost copybook mastery without restoring the library. Gentler pressure was the only dial that honestly helped.

Mechanism, measured: weight movement (L2 distance from base) saturates within the standard run — 97% of all movement at 30× happened by step 300 — because the answer-set loss hits 10⁻⁴ and gradients vanish; base-corpus loss rises just 3% more from 1× to 30×. So "overfitting destroys the base model" is the wrong headline at fixed lr; the right ones are "fine-tuning itself costs a quarter of the library in its first 25 steps and half by step 50" and "lr multiplies weight displacement" (2.4× at lr 0.05, taking retention to 13%). The one genuinely monotone overtraining effect: verbatim-parroting of demonstrated answers to fresh questions roughly doubles on average from 1× to 30×, and rises in 10/10 seeds. The library-flatness and parroting-climb halves of this paragraph re-run live in the panel above — the 10× button is the actual experiment in your tab; the weight-norm, base-corpus-loss, and 30× figures are from the 10-seed repo lab.

Everything you just did to the little reader, real AI labs do to the giants. The copybook is bigger — thousands of beautifully written question-and-answer conversations — and the reader has read the whole internet instead of one storybook. But the trick is the same trick: show, trace, repeat.

And the problems you found are their problems too. Real assistants also answer when they shouldn't. Real assistants also forget a little of what they knew. The people who build them spend enormous care on exactly the things you just watched go wrong.

The scale jump: our copybook had 16 demonstrations; InstructGPT used about 13,000; Alpaca, 52,000 (written by a bigger model — distillation in a chat costume); LIMA, famously, just 1,000 very good ones. The consistent lesson across all of them: quality beats quantity, because SFT is teaching a habit, and habits are learned from clean examples. A thousand perfect demonstrations teach a cleaner habit than fifty thousand sloppy ones.

And one more thing SFT quietly teaches that's easy to miss: when to stop. Every demonstration ends; the model learns the ending mark like any other habit. A chatbot that knows when it's done talking learned that from its copybook too.

The modern SFT stack, condensed: a chat template with role markers and end-of-turn tokens (get it wrong at inference and behavior falls apart — template mismatch remains a top real-world bug); prompt masking as the default loss configuration; data curation as the dominant cost center — dedup, decontamination against benchmarks, mixing ratios across task types, and replayed pretraining data to blunt forgetting; parameter-efficient variants (LoRA and friends) that freeze the base and train low-rank adapters, trading a little ceiling for a lot of memory; and model-written demonstrations — constitutional-AI-style self-critique, or teacher-model generation à la Alpaca — which is where this page's Part 2 (distillation) and Part 3 formally merge: sequence-level distillation and SFT are the same operation with a different data source.

Where the ceiling is: SFT is behavior cloning, so three limits are structural. (1) The demonstrator caps the policy — the model imitates labelers, including their mistakes and styles; it cannot exceed them except by averaging. (2) Teacher forcing means no training signal on the model's own rollouts — exposure bias, compounding at generation time. (3) Maximum likelihood is mode-covering over demonstrations and blind to the model's own knowledge state — the hallucination mechanism you measured. All three point the same direction: past a competence threshold, examples are a weaker signal than judgment on the model's own outputs — which is why every frontier pipeline hands off from SFT to preference optimization and RL, keeping SFT as the reference policy π_ref that RLHF's KL leash is anchored to. The handoff, and what judgment fixes that copying cannot, is the companion page: The Machine That Learns From Consequences.