Knowledge Distillation
Knowledge distillation is a technique for training a smaller "student" model to imitate a larger "teacher" model, so it keeps much of the teacher's ability at lower cost.
What is Knowledge Distillation?
Distillation is widely used to build fast, cheap models. Unauthorised distillation from a competitor's model outputs is a growing commercial and legal concern.
How it works
The student is trained on the teacher's outputs or probability distributions rather than only on original labels.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions