⭐️ Fine-Tuning vs. Distillation vs. Quantization, Explained with LEGO

Fine-tuning, knowledge distillation, and quantization are often grouped together, but they solve different problems:

• Fine-tuning specializes a model with targeted data.
• Distillation transfers useful behavior to a student model, often a smaller one.
• Quantization stores the model’s numbers at lower precision.

Here is an explainer video, explained with LEGO:

The short version: specialize, teach, lower precision

That gives us a useful three-word mental model:

Fine-tune to specialize. Distill to transfer. Quantize to use less precision.

Fine-tuning: specialize the model

A pretrained model has broad capabilities. Fine-tuning continues training on a smaller, targeted dataset to improve a task, domain, style, or output format.

In the LEGO analogy, the green model becomes a red specialist. Its size stays the same because full fine-tuning changes weight values, not necessarily the architecture or parameter count.

Methods such as LoRA usually freeze the base weights and train a much smaller set of adapter parameters. The goal is still specialization, but the adaptation is cheaper to train and store.

Distillation: train a student from a teacher

In knowledge distillation, a capable teacher provides training signals for a student. The student learns to reproduce useful parts of the teacher’s behavior, often with a smaller, cheaper architecture.

Distillation does not delete pieces of the teacher or convert it into a smaller file. It trains a separate model that must be evaluated on the behavior that matters.

PNG version

Quantization: represent numbers with fewer bits

Model weights are numbers, often stored as 32-bit or 16-bit floating-point values. Quantization represents weights-and sometimes activations-with lower-precision formats such as 8-bit or 4-bit values.

For a 1-billion-parameter model, the idealized weights-only memory falls in proportion to bit width:

PNG version

Fewer bits can reduce memory, storage, bandwidth, and inference cost on compatible hardware. The tradeoff is approximation error, which may affect quality.

One caveat: the quantized LEGO model appears narrower, but quantization usually does not remove parameters or change the architecture. The shape represents a smaller memory footprint-the same collection of values stored with fewer bits.

How to choose

Choose based on the outcome:

• Fine-tuning: change behavior, task performance, style, or output format.
• Distillation: move useful behavior into a different, often smaller, model.
• Quantization: reduce memory, latency, or deployment cost when the model is already capable enough.

The techniques can also be combined. A production workflow might:

1. fine-tune a teacher for a specific task;
2. distill its behavior into a smaller student; and
3. quantize the student for deployment.

PNG version

Where the LEGO analogy stops

LEGO makes the differences visible, but real systems add nuance:

• Fine-tuning can update all weights or only a small set of adapter parameters.
• Distillation can use logits, generated examples, intermediate representations, or other signals.
• Quantization can be applied after training or incorporated into training.
• Smaller or lower-precision models are not automatically faster; hardware and software support matter.
• Every technique requires evaluation for its own quality tradeoffs.

Use the analogy as a map, not the territory.

Further reading

• Hugging Face: Fine-tuning
• Hinton, Vinyals, and Dean: Distilling the Knowledge in a Neural Network
• Hugging Face: Quantization

What should I explain with LEGO next: LoRA, pruning, or speculative decoding?

LEGO® is a trademark of the LEGO Group of companies, which does not sponsor, authorize, or endorse this content.