Self-distillation: DINO, DINOv2 and DINOv3 · MAAI7602 Applied Computer Vision

DINO trains a student network to match the output distribution of a teacher that is a moving average of the student. Centring and sharpening of the teacher prevent the two kinds of collapse. How does DINO train a network to match a teacher built from itself, and how is collapse avoided?

This page needs JavaScript for its explanations, activities and practice questions.