What is AI Model Distillation? -what does it involve

What is distillation

Model distillation is cloning the behaviour, but not the model itself, of a larger model to create a smaller, cheaper, not quite as good model by training it from the larger fully formed model. This is done by training the smaller model from the response to stimulus of the larger model. It is much cheaper to train a model by distillation versus the full training process from scratch. The resulting model can be "good enough" for many applications.

There are different approaches, and choice of approach depends on the reasons behind the distillation and the depth of access to the source model. It is possible for one company to clone the way another company's model behaves using distillation.

Distillation is like the training of junior doctors, or apprenticeships. A consultant surgeon will often train a junior surgeon. The consultant is smart and very experienced but also expensive and can be burdened by all the experience and knowledge, being slow due to other teaching responsibilities and other research activities they have due to their senior position. Rather than handing the junior some notes and crossing your fingers wishing them good luck, instead we make them shadow the consultant on hundreds of cases. The junior will observe the presenting patient problems, then see the decisions the consultant makes and maybe hear the consultant explain the thinking and decision process on why they take the action they do in each case. They will practice together until their judgment and decisions align. Note that the junior surgeon will not match the senior for the rarest, most difficult cases, due to not actually having spent a lifetime working together, but for the vast majority of everyday work, they will do just as good as the consultant and be far cheaper.

Essentially, that is the distillation process. A large, capable LLM model is fed a wide variety of stimulus (work), and it generates output. The smaller LLM model is trained on that input and output, to give the same behaviours. This results in the new model being smaller and cheaper to run and in most cases "good enough" in the output it produces, but it is a surface clone, it does not have inside the model the same complexity of probabilities driving it as the original.

Sometimes, if the appropriate level of access to the original model is available, essentially access to the probabilities that derived the output, then it becomes possible to also distil some of the "thinking" process too, rather than just copying the behaviour.

The process involves literally throwing large volumes of "questions"/work at the source LLM and capturing the answers.

What are the benefits?

A distilled model can be lighter weight, it does not have such a big set of probabilities inside driving it, you are effectively compressing the big model into a smaller model, losing the detail but still good enough for many purposes. Smaller models are cheaper to run than big ones, quicker too, less resource hungry, so faster responses, cheaper inference, and the ability to run on smaller or less powerful devices. The model can also be specialised, trained on a subset of knowledge, making it quick and satisfactory at a specific task.

Inference also explains why you see frontier labs release mini models alongside the big models. These are often the distilled (compressed) version of the big parent model.

But how is it actually done?

The way distillation is done varies depending on how much access there is to the source model's internal probabilities. One lab can clone another's model, in that case they may just be using a public API to clone the model and will not have any privileged access. On the other hand, a lab may be training the "mini" version of their own full model. In that case they are able to train with a lot more literally "insider knowledge".

1. Learning responses (common)

Going back to our surgeons, this is where the junior reads thousands of the consultant's case notes: "Patient A presented with this and we did that." They don't get to experience the surgery, just see the input, action and outcome.

In this case, the main model has a large, diverse set of prompts thrown at it, the responses are collected, and those pairs of question and response become the training data for the smaller model. The smaller model is fine-tuned with that data set, just the same as with any other dataset.

Sometimes the prompt will ask for the reasoning too, this reasoning trace can also be added to the question and response, to help the smaller model pick up how the answer is derived too.

This method is popular, as it's just text, you can do this against any architecture and it scales well, as text is just inference cost. It also works through a closed API, this is what has made the news in the past when other nations or companies "steal" model behaviours this way. You don't need much access to the source to clone it.

The downside is that the lack of depth can mean the smaller model ends up producing similar sounding results, but without the quality of thinking or with limited capability.

2. Degrees of confidence (classic)

Going back to our junior doctor: if, when reading the previous case notes, they also read the extended notes that include where the consultant noted the percentage certainty they had of the condition being different things 70% likely pneumonia, 25% bronchitis, etc. This provides the smaller model with much more representative data to learn from when training. It gives the detail that fed the reasoning.

For this approach the temperature of the model is changed (how random it is), and the smaller model is trained on the full statistical probability distribution of different responses. This gives the learning model more sight of what other answers the "consultant" might have also given, which helps the statistical insight it needs to properly clone its behaviour. The small model will better align with the large model's statistical distribution.

This does, however, require direct access to the raw internal probabilities of the source model, not something you get access to from a public API. The two models also need to have the same vocabulary, so they need to be similar model families.

3. Learning underlying instincts (research)

This is like using biosensors linked up to the surgeon's hands and eyes to actually record the actions they make during surgery. This allows the junior doctor to later replay exactly how the consultant moved to do the surgery, letting them learn those exact movements themselves, which is a whole other level of data.

The small model is trained so its internal calculations/movements are like the large model's at the same corresponding points. This is a much deeper level of computer science and I will leave it there, as I am out of my depth now. My neural networks and computer vision modules at university only took me so far.

It requires deep access to the model's internal workings, and both small and large models need to be built on the same technology to be compatible. This is mostly used in research settings.

Distilling a rival model

DeepSeek/OpenAI, back last year, was a case in the news where a company was accused of distilling a competitor's model without permission and against the terms of service of the contract.

The only method available to a rival is method 1, they will:

1. Query the target model with a lot of data using the public API or chat interface. No special access is required.
2. Log responses, including the reasoning traces if they can be exposed.
3. Fine-tune the copy model using the obtained data.

Anyone with an API key and a big budget can extract training data.
There is no protection that can prevent this at the API level. They are just using the model, sending text in, getting text out. Usage monitoring and output-pattern detection can work to try to detect it, but it is difficult to distinguish from other workloads.