The machine that teaches machines

A caterpillar has one job: eat. A butterfly has a different job: fly. Same animal — two shapes, one for gathering and one for going places.

Big AI models are caterpillars. They live in warehouse computers and munch through nearly the whole internet. But you can’t fit a warehouse in your backpack. So scientists learned to turn the caterpillar into a butterfly: a much smaller model that keeps most of what the big one learned. The trick is called distillation — named after the way you boil a big pot down to a few precious drops.

You already used a butterfly today. When your phone guesses your next word, or your camera translates a menu with no internet — that’s a tiny model, flying on knowledge a giant gathered.

The models behind modern AI are enormous — up to hundreds of billions of adjustable weights, running on racks of data-center GPUs. Fine for a chat website; useless inside earbuds, a car’s lane camera, or a clinic laptop with no internet — places that need answers in milliseconds, on a battery, in private.

The odd part: a model’s size matters most while it learns, not while it answers. Hinton, Vinyals and Dean opened the famous 2015 distillation paper with exactly this picture — an insect is a larva when its job is extracting energy from food, and a lighter, completely different form when its job is traveling. Train a giant; deploy a butterfly.

Does knowledge survive the squeeze? DistilBERT (2019) kept about 97% of BERT’s score on the GLUE language benchmark with 40% fewer weights, answering 60% faster on CPU. And in 2025, DeepSeek used its giant reasoning model R1 as a teacher for students as small as 1.5 billion weights — small enough for a gaming PC. (That story gets its own section, Part 7.)

The idea is older than its fame: back in 2006, Buciluă, Caruana and Niculescu-Mizil squeezed a whole committee of models into one and called it “model compression.” Hinton’s paper gave it its modern form — and its better name.

Deployment lives under a three-way constraint: latency (autoregressive decoding is memory-bandwidth-bound; a 70B streams tokens roughly an order of magnitude slower than an 8B), cost (per-query FLOPs × fleet size), and placement (on-device for privacy and offline). Distillation is one tool in the compression toolbox, and it composes with the others:

Hinton’s original target wasn’t even a single big model — it was an ensemble. A committee of specialists is accurate but absurd to serve; distill the committee’s averaged predictions into one model and serve that. The same logic runs through today’s pipelines: Gemma 2’s smaller models (2B and 9B) were trained with distillation from a larger teacher’s token distributions, and NVIDIA’s Minitron line prunes a big model, then distills to recover the lost quality.

When is distillation compute-optimal versus simply pretraining the small model? Distillation scaling laws (Busbridge et al., 2025, arXiv:2502.08606) map the frontier: when a teacher already exists — or its cost amortizes across many students — distillation dominates direct pretraining up to a student-compute budget, beyond which direct pretraining wins; and an over-strong teacher can actively hurt (the capacity gap, Part 7). The mechanism-level reading: soft targets buy per-example information efficiency, which matters exactly when student data or compute is the binding constraint.

One mechanism, three uses — and this page walks all three. Compression (this part) is the famous one. But matching soft targets also acts as a regularizer that can lift a student the same size as its teacher (Parts 3–4), and looped back onto a single model it becomes a self-improvement — and self-collapse — machine (Part 5). Keep the triptych in mind; conflating the three is the most common way to over-claim what distillation does.

Scroll back up to the drawing pad for a second. About your squiggle, the answer key says one thing: SEVEN. The teacher says more: “mostly seven… a little bit one… definitely not a zero.”

That extra part is like getting your quiz back from a teacher who wrote “B was a smart guess!” next to the ones you missed — instead of just a big red ✗. You learn from the almost-right answers, not only the right one.

Scientists call this hidden lesson dark knowledge — like dark matter in space: you can’t see it in the final answer, but most of what the model knows is stored there. Distillation is simply letting a small model study the teacher’s “how close was each answer” scores instead of the bare answer key.

A classifier doesn’t really output an answer; it outputs a probability for every class. The 2015 paper’s own examples: a handwritten 2 that gets a one-in-a-million chance of being a 3 but only one-in-a-billion of being a 7; a photo of a BMW with a tiny chance of “garbage truck” and almost none of “carrot.” Those tiny numbers encode the similarity structure of the world — which mistakes are reasonable and which are absurd. A one-hot label erases all of it.

There’s a catch: a well-trained teacher is confident, so on most inputs the interesting numbers hide close to zero — a clean 7 from this page’s test set reads over 99.9% “seven,” with the whole ranking buried in the crumbs. The fix is the slider you already played with: temperature. Divide the model’s raw scores by T before turning them into probabilities, and the faint ranking swells into something a student can actually learn from. And a few digits carry their dark knowledge loudly even cold — like a 7 the teacher reads as only 53% seven, 25% one. Three of the loudest live in the gallery below.

So the recipe is: warm up the teacher’s answers, and train the student to match them — usually alongside the true labels. The name dark knowledge comes from a talk Hinton gave in 2014; the method from the 2015 paper; the seed of the idea from Buciluă and colleagues in 2006.

The KL term’s gradient (before its T² multiplier) with respect to a student logit is (1/T)·(p_s,i − p_t,i) — and for a fixed logit gap, that probability difference itself shrinks like 1/T. So the raw soft-target gradients scale as ~1/T², and the T² factor exists for exactly one stated reason: it keeps the relative weight of the hard and soft losses roughly constant while you tune T. It decouples the two knobs. Bookkeeping, not magic.

In practice T ≈ 2–5 is the usual band for classification, with most of the weight on the soft term. And logits aren’t the only thing worth matching: FitNets match intermediate activations (“hints”), attention transfer matches attention maps, and DistilBERT’s actual objective is a triple — soft targets + masked-LM + cosine alignment of hidden states.

Convention notes worth pinning: the soft term as written is forward KL, which equals cross-entropy with soft targets up to the (constant) teacher entropy; repositories disagree about where the T² lives; and reversing the KL direction is not a convention choice but a different objective — mode-seeking rather than mode-covering. That distinction becomes load-bearing for language models in Part 7.

Time for a real experiment — it runs right now, in your browser. Two robot students with exactly the same brain want to learn to read handwritten digits. Each gets the same 60 flashcards, made by the teacher you met in the drawing pad.

One difference. Student A’s flashcards just say the answer: “this is a 7.” Student B’s flashcards show the teacher’s whole feeling: “mostly 7… a little 1… definitely not 0.” Same cards, same practice time. Make your guess in the experiment below, then watch them train — for real, live — and see whose line climbs higher.

Same network, same 60 transfer digits, same optimizer, same number of steps — the only difference is what’s written on the labels. Student A gets the teacher’s answer key (one-hot: the winning digit, nothing else). Student B gets the teacher’s full softened beliefs. Could the labels alone matter? Place your bet below, then watch two real networks race — every press trains them from scratch, live in this page.

The setup, fully seeded and reproducible. Both students are the same 64→24→10 ReLU network (1,810 weights) started from the same He-initialized draw; 60 transfer digits sampled from a 600-digit pool; full-batch SGD, momentum 0.9, lr 0.3, 1,500 steps. Student A’s targets: one-hot argmax of the hero’s 3,610-parameter teacher (97% test accuracy). Student B’s targets: the teacher’s softmax at T = 2, trained with the matching T²-scaled gradient (dz = T·(σ(z/T) − q)/B). The curves are accuracy on a fixed 200-digit held-out subset, evaluated as training runs.

One fairness check worth making before you bet: the hard student is not sandbagged by sharing the soft student’s hyperparameters. In offline validation with this exact configuration, giving the hard student its own best learning rate from a 7-point sweep (0.05–0.5) doesn’t save it — it still loses on 10 of 10 seeds, mean gap ≈ 6.3 points and never under 1 (at the shared lr the mean gap is ≈ 7.5). Whatever the curves show, it isn’t a tuning artifact.

Student A stops learning almost immediately. Not because it’s dumb — because its flashcards run out of things to say. “It’s a 7” is one fact. “Mostly 7, a little 1” is a fact plus a map: it says which digits look alike. Student B keeps climbing because every card teaches the map, not just the answer.

Why does the soft student keep climbing after the hard one flatlines? A one-hot label carries at most log₂10 ≈ 3.3 bits — “it’s a 7, full stop.” The teacher’s softened distribution carries its whole similarity map between digit classes: this 7 leans 1, that 8 is half a 2, 4s and 9s rhyme. With only 60 examples the answer key is exhausted within the first hundred steps — the hard student plateaus around 80% and stays there. The soft labels keep paying out, into the low 90s. The disagreement strip shows where the difference lives: mostly the borderline digits.

That’s the honest core of distillation’s advantage: information per example. Give the hard student ten times the flashcards and it would mostly catch up — soft targets matter most when the student can’t see much data.

Details worth noticing. The teacher’s dashed line is drawn at its accuracy on the same fixed 200-digit eval subset the curves use — 98.0% there, 97.0% on all 300 test digits — and it stays out of reach for both students; this is compression, not magic. The hard student’s plateau is an information ceiling, not an optimization failure: its training loss keeps falling (it fits the 60 one-hot targets essentially perfectly) while test accuracy stalls — it is memorizing the last few cards, not learning the map. The T = 2 softening matters too: at T = 1 the teacher is so confident that most of the map hides in the last percent of probability. Automated check: window.__raceBench(3) replays the race on 3 seeds and reports wins and gaps.

Two honest counterpoints belong next to this demo. Yuan et al. (2020): much of KD’s benefit in the wild is generic regularization — even teacher-free or bad-teacher distillation helps — and in a non-convex student like this one, regularization and information transfer are entangled; the design here (shared init, shared data, shared schedule, labels as the only difference) removes the usual confounds but cannot fully separate those two readings the way a convex toy could. And Stanton et al. (2021): students routinely fail to match their teachers even when they generalize better — look at the final curves: the soft student beats the hard one without coming close to the teacher’s line. Read the race as strong evidence for the label channel, not as a theorem.

Ready for a magic trick? We train a brand-new student — but first we steal every 3 from its flashcards. It grows up never seeing a single 3. Not one.

Then we hand it thirty 3s and say: read these. Could you read a letter nobody ever taught you? Make your guess below — then try all three ways of teaching, and watch the trick happen (or not) in front of you.

This mirrors the boldest experiment in Hinton’s 2015 paper: delete a whole digit class from the transfer set, distill, then test on the deleted class. Our version: the transfer set is the 540 digits of the pool that aren’t 3s. Three students train on it — one on the teacher’s hard answers, one on its softened beliefs at T = 2, one at T = 6 — and each is then shown the 30 held-out test 3s. The only place “3” exists in their training is inside the teacher’s doubt about other digits: the faint “…and a little 3-ish” it leaks on certain 8s, 9s and 5s. Bet first; then climb the ladder.

Config: student 64→32→10, He init, SGD momentum 0.9, lr 0.05, minibatch B = 100 (seeded shuffle per epoch), 2,000 steps, transfer = all 540 non-3 pool digits, targets = teacher softmax at the rung’s temperature (or its one-hot argmax), scored by raw argmax on the 30 test 3s — no bias correction. One honest calibration note: on these 8×8 digits T = 1 is genuinely too cold — the 3-logit finishes around rank 7 of 10 and scores 0/30, exactly like the hard rung — so the ladder’s middle rung is T = 2, the lowest temperature that reliably lifts the 3-logit into the top two (top-two on ~25 of 30 test threes; outright top on roughly half). The bias slider below reproduces the paper’s garnish: add b to the 3-logit at evaluation time and watch the trade against the other nine classes.

Why is the hard rung not just bad but stuck at zero? Every one-hot target sets p(3) = 0, so training monotonically starves the 3-logit — emitting 3 is pure loss everywhere on the transfer set. The soft rungs invert that: at T = 6 the teacher’s mean softened p(3) across the 540 non-3s is ≈ 3% — small but structured, concentrated on 3-ish digits — enough supervision to aim the 3 direction in weight space without one positive example. The label-free way to choose the bias b, if you want one: raise b until the student predicts “3” at roughly the 1-in-10 prior rate. For calibration: Hinton’s MNIST version was already strong before that garnish — the distilled model read 877 of the 1,010 test 3s (≈87%) raw, and 98.6% after raising the 3-logit by 3.5 (Hinton, Vinyals & Dean 2015, §3). Our warm rung reaches ≈90% raw with no bias at all — the honest contrast is the garnish budget, not the raw score: 8×8 digits let temperature do the whole job.

So a student can know things nobody wrote on any flashcard. Here’s the stranger part: sometimes a student taught from another robot’s doubt ends up better than its teacher. Like recopying your class notes — the copy often comes out cleaner, because you keep the important parts and drop the scribbles. This year’s student really can become next year’s teacher.

But hold on to the word sometimes. In the next part, you’ll see what happens when you photocopy the photocopy, again and again…

Born-again networks (Furlanello et al., 2018) run this logic at full scale: train a network, freeze it, train an identical twin from scratch to match its soft predictions — and the twin comes out consistently a little better than the original on benchmarks like CIFAR and language modeling. Third and fourth generations keep helping, less and less. The headline-grabbing cousin is Noisy Student (Xie et al., 2020), which pushed ImageNet to 88.4% — but honest label: that one is self-training, with an equal-or-larger student, 300 million extra pseudo-labeled images and heavy noise. Soft targets were one ingredient, not the whole cake.

And the honesty note this page owes you about its own demo: Hinton’s original MNIST run was already startlingly good raw — the distilled model read about 87% of the deleted 3s before any correction, and 98.6% after a small bias fix on the 3’s output. Our 8×8 version reaches roughly 90% raw and needs no bias fix at all; the middle rung of the ladder shows how much of that the temperature knob alone buys. Same phenomenon, friendlier stage — the honest difference is the garnish: their headline needed the bias nudge, ours doesn’t.

Why would copying a teacher’s outputs ever beat training on the truth? Soft targets look like label smoothing, a known regularizer — but they are instance-specific: this particular ambiguous 8 is half a 2; that crisp one isn’t. Uniform smoothing carries zero bits about the input; dark knowledge carries the teacher’s whole similarity map. The interplay has its own research thread: Müller et al. (2019) found that teachers trained with label smoothing are better calibrated yet distill worse — smoothing collapses the intra-class structure living in the logits. Shen et al. (2021) contested how general that is at scale; treat it as a live debate. What is settled is a recipe: “a good teacher is patient and consistent” (Beyer et al., 2022) — show teacher and student the same augmented view, train far longer than feels reasonable. Still the strongest plain-KD recipe in vision.

Two families of theory compete to explain same-size gains. The regularization view: Yuan et al. (2020) showed teacher-free KD — distilling from a hand-designed smooth distribution, or from a much worse teacher — still helps, so part of KD’s benefit is generic output-space regularization. The ensemble view: Allen-Zhu & Li (2023) model data as carrying multiple predictive “views”; independently trained networks latch onto different subsets, and distillation — including self-distillation, read as implicit ensemble-then-distill — transfers the union. The missing-digit ladder sits squarely in the information camp: no generic regularizer can conjure a class absent from the data; only the teacher’s structured doubt carries it.

And the honest boundary: in pure compression settings, students essentially never surpass their teachers — R1-Distill-32B is remarkable for its size; it does not beat R1. Surpassing needs same-or-larger capacity, extra data, or both.

Everyone knows what happens when you photocopy a photocopy: the page gets uglier every time. Let’s test that on robots. The teacher trains a student. That student — without ever seeing the teacher again — trains the next student. And so on, eight copies deep.

The fun part: this photocopier has three settings. Copy the doubt. Copy only the answers. Or send the page through smaller and smaller copiers. Bet first: after 8 doubt-copies, how much reading skill is lost?

The chain: teacher → S1 → S2 → … → S8. Each generation trains only on the previous generation’s outputs over the same transfer digits — after the first hand-off, the original teacher and the true labels are gone from the loop forever. Three modes: soft copies (each hand-off passes beliefs at T = 4), one-hot copies (each hand-off keeps only the winning digit), and shrinking copies (soft hand-offs, but the students step down in size — the first matches the teacher’s parameter count, and every one after is smaller than the last). Everyone “knows” copies of copies degrade. Place your bet, then run the chain and watch the accuracy line — and the six borderline digits below it.

Config: soft and one-hot generations are 64→32→10 (fresh seeded init per generation), M = 400 pool digits, B = 100, 1,000 steps; lr 0.1 with T = 4 targets for soft, lr 0.3 with argmax targets for one-hot. Shrinking mode walks hidden sizes 48→32→24→16→12→10→8→6 over the full 600-digit pool, 2,000 steps, lr 0.1, T = 4. Accuracy is scored on all 300 held-out test digits. Disclosure: the default seed (4) was picked for drama — its shrinking chain loses ≈13 points where nearby seeds typically lose 3–9, and one validated seed (7) collapses below 70%. New seed shows you the distribution; the caption always reports the run you actually got. The Engineer chip tracks mean confidence vs accuracy — watch it in the lossy modes: confidence holds near 98% while accuracy falls. Overconfident copies are the signature of information bleed.

The one-hot chain is the clean information-theoretic case: the first argmax hand-off sets target entropy to zero, so whatever dark knowledge the teacher carried is unrecoverable from generation 1 onward — later generations copy an already-blunt world, and the chain goes flat rather than continuing to decay. The shrinking chain is the capacity story: each generation preserves the similarity map as well as its width allows, and the bleed compounds. Note what does not happen at this scale: the soft same-size chain does not collapse. That absence is a finding, and the demo is built around it.

The doubt-photocopier barely loses anything — eight copies later, still reading fine. The answers-only photocopier ruins the page in one copy, then faithfully re-copies the ruin. And the shrinking copiers leak a little every time, without ever noticing. The lesson of the whole page, one more time: the doubt is the knowledge. Copy it and the knowledge survives. Drop it, and one copy is all it takes.

The debrief. Soft copies: the chain dips a couple of points at the first hand-off, then holds — ending within one to three points of the teacher on every seed we validated. At this scale, soft self-distillation is a nearly lossless photocopier, and that is the punchline; no secret collapse was hiding in the fine print. One-hot copies: the damage all happens in generation 1 — the moment the doubt is flattened to a single answer, the similarity map dies, permanently; later copies wobble around that blunted level without ever approaching the teacher again. Shrinking copies: a bleed that compounds as the map is squeezed into fewer and fewer neurons — a few points on most seeds, double digits on unlucky draws (the page’s default seed is deliberately one of the dramatic ones; New seed shows you typical runs) — while every generation stays as confident as the first. Confidence above accuracy, drifting apart: that mismatch is how you detect a decaying chain from the inside, without labels.

The debrief, with the diagnostic. Soft copies dip ≈1–2 points at the first hand-off and then hold, ending 1–3 points under the teacher across validated seeds — no collapse at constant capacity. One-hot copies pay their whole toll at generation 1, where the argmax hand-off zeroes the target entropy; after that the chain just re-fits a blunter function, wobbling a point or two either way. Shrinking copies are the compound-interest case, and the chip to watch is confidence vs accuracy: mean top-1 confidence stays in the high 90s while accuracy slides — on lossy chains the two drift apart, and that gap is a label-free decay detector you can use on real pipelines. Seeds differ widely in how hard the squeeze bites (≈3 points up to ≈30 on the worst validated draw — and the default seed was picked for drama, so press New seed); the caption under the chart always describes the run you just watched. Replay any chain headlessly with window.__chainBench('soft'|'onehot'|'shrink', seed).

The exact-math version of this chain lives in kernel regression: Mobahi, Farajtabar & Bartlett (NeurIPS 2020) prove that ε-budget self-distillation in an RKHS multiplies a spectral filter onto the solution each round — regularization amplifies, a few rounds help a deliberately overfit model, and the chain provably collapses to the zero function in finitely many rounds. An earlier version of this page ran that construction live; this one demonstrates the phenomenon on real data instead, where the honest headline changes: with soft targets and constant capacity the chain is nearly lossless, and knowledge only dies through a lossy hand-off — argmax or shrinking width. The theorem and the demo agree on the mechanism (every hand-off filters); they disagree, instructively, on how fast the filter bites. Both readings of self-distillation theory remain live: regularization amplification, exact in the RKHS, and Allen-Zhu & Li’s implicit ensemble-then-distill for deep networks.

Here’s a secret about Part 1 of this series — the page about machines that teach themselves. The guessing games there had a teacher all along. It was the same robot: a calmer, steadier copy of itself.

Imagine the you from last week grading today’s homework. Last-week-you isn’t smarter — but they’re steadier, an average of many days of you. It turns out that matching a steady copy of yourself is a great way to learn, with nobody else in the room.

In BYOL and DINO — Part 1’s “escape artists” — the student network updates by gradient descent, while the “teacher” is an exponential moving average of the student’s own past weights. The student sees one augmented view of an image and must match what the teacher says about another view. No labels anywhere in the loop.

Read that with this page’s eyes and it names itself: a student matching a teacher’s outputs is distillation, and a teacher that is your own smoothed history makes it self-distillation. Meta’s DINO wears the fact as its name: self-DIstillation with NO labels. If you read Part 1, you had already met this chapter — it just wasn’t wearing the costume. Revisit Part 1 with the dial set to Student and see for yourself.

Mechanics worth keeping straight. The EMA momentum is high (≈0.99–0.999), so the teacher trails the student by a long, smooth window. In DINO, teacher outputs pass through a softmax at temperature ≈0.04–0.07 while the student uses ≈0.1 — the teacher is sharpened relative to the student — plus a running-mean centering of teacher logits; sharpening alone would collapse the outputs onto one dimension, centering alone toward uniform, and the pair holds the balance. BYOL, by contrast, has no softmax and no temperature at all: it is a cosine regression between the student’s predictor and the teacher’s projection, with a stop-gradient. So this page’s temperature intuition maps onto DINO — not BYOL.

Self-teaching loops fail in characteristic ways, and it pays to hold two collapse stories side by side. DINO-style collapse: teacher outputs drift toward a constant or uniform distribution and the student happily matches it — a degenerate fixed point of the matching game, killed by centering + sharpening keeping the output distribution both spread and committed. Mobahi-style collapse (the exact-math aside in Part 5 — you watched its practical cousins in the one-hot and shrinking chains): each ε-budget self-distillation round provably amplifies regularization until the fit snaps to the zero function. Different mechanisms; same moral — a self-referential teacher must be paired with a force that keeps the signal alive.

There is also a satisfying ensemble reading of the EMA teacher: a moving average over the student’s trajectory behaves like a cheap ensemble of past selves — which quietly connects the momentum-teacher trick to the Allen-Zhu–Li account of why distillation transfers anything at all.

Today’s biggest AIs use distillation in two fun ways.

The intern and the boss. A small, quick model drafts the next few words. The giant model checks the whole draft in one glance and keeps the parts it agrees with. The final answer is exactly what the boss would have written — just two or three times faster. (The grown-up name is “speculative decoding.”)

Homework from a giant. A giant “reasoning” model writes out its thinking, step by step, hundreds of thousands of times. A small model studies those worked examples like a student with a solutions book — and learns to think out loud too. That’s how the small reasoning models that run on a laptop were made.

There are two ways to learn from a giant, depending on what you can see. Black-box: you only see its final text. Collect its answers and fine-tune on them — that’s how DeepSeek turned R1 into a family of small “distill” models: roughly 800,000 R1-written worked examples, then plain supervised copying. No reinforcement learning at all — just a very good solutions book. White-box: you can see the teacher’s full probability list for every next token — far richer signal per example; Google trained Gemma 2’s 2B and 9B models this way from a larger teacher.

Speculative decoding deserves its own box, because it is distillation’s mirror image: instead of making a lossy copy, you keep both models. The small one drafts, the big one verifies in parallel, and a clever accept/reject rule guarantees the output distribution is identical to the big model’s. Lossless, typically 2–3× faster — and the best draft models are, of course, often distilled from the very boss they draft for.

Token-level KD matches per-token distributions; sequence-level KD (Kim & Rush, 2016) fine-tunes on the teacher’s generated outputs — R1-style trace training is sequence-level KD at scale. Both inherit a classic flaw: the student trains on the teacher’s prefixes, but at inference it must continue its own — the distribution mismatch known as exposure bias.

The fix is to go on-policy: let the student generate, and have the teacher grade the student’s own tokens. GKD (Agarwal et al., 2024) does exactly this. MiniLLM (Gu et al., 2024) additionally swaps the objective to reverse KL — mode-seeking: a small student concentrates on what it can actually do well, instead of smearing probability across every mode of the teacher the way forward KL’s mode-covering does (which, under a capacity limit, produces mush).

Failure modes to engineer around: the capacity gap — a bigger, more accurate teacher often yields a worse student (Cho & Hariharan, 2019; early-stopped teachers distill better). Teacher-assistant chains (TAKD, Mirzadeh et al., 2020) bridge the gap in some setups, though Cho & Hariharan found sequential chains ineffective — the evidence is mixed. And the fidelity gap (Stanton et al., 2021): students routinely fail to match teacher predictions even when their generalization improves — optimization, not capacity, is often the real bottleneck. One more composition: DistillSpec distills the draft model toward its boss to raise speculative acceptance rates — the two ideas stack.

The scaling-law lens (Busbridge et al., 2025) turns folklore into curves: at fixed student compute there is a weak-teacher regime where distillation beats direct pretraining and a strong-teacher regime where the capacity gap bites — and whether the teacher’s own training cost counts (or is amortized across a model family) flips the verdict. This is why production families (Gemma, Llama-3.2’s 1B/3B via pruning+distillation, Minitron) treat one big run as a tree of distilled offspring.

Position on-policy distillation on the post-training map and its appeal is obvious: SFT is off-policy with dense supervision; RL (RLHF/RLVR) is on-policy with sparse rewards; on-policy distillation is on-policy with dense per-token supervision — often the most compute-efficient way to move capability into a small model, and increasingly the default recipe. One legal footnote belongs in any honest survey: distilling from a commercial API’s outputs may violate its terms of service — a live policy debate, separate from the science.