Could On-Policy Self-Distillation Revolutionize Post-Training?

Foundation models often suffer from "catastrophic forgetting," a phenomenon where learning new skills causes them to lose previously acquired abilities. A new technique called On-Policy Self-Distillation (OPSD) offers a promising solution by enabling models to learn continually without this performance degradation. This method involves a model teaching itself, using a "student" and a "teacher" copy of the same network. The teacher is given access to privileged information, like expert demonstrations, and the student is trained