Making Knowledge Distillation Cheap Enough to Run at Scale

Knowledge distillation, a technique to train smaller student models to match larger teacher models, is gaining renewed interest with open-source Large Language Models like gpt-oss, Qwen, GLM, and Kimi. Deploying massive models such as the Kimi-K3 (2.8 trillion parameters, 3TB VRAM) is costly. Compressing these into smaller, capable models via distillation is now standard, with companies like Nvidia (Nemotron 3 Puzzle 75B) and Multiverse Computing (Hypernova 60B) releasing compressed models. The chunked-loss implementation is open-sourced at github.com/CompactifAI/Full-Chunked-KL-Loss.