Reverse Distillation: Consistently Scaling Protein Language Model Representations
Department of Computer Science(计算机科学系);Duke University(杜克大学);Center for Computational Biology(计算生物学中心);Flatiron Institute(Flatiron研究所);Depts of Biostats. and Bioinformatics(生物统计学与生物信息学系;细胞生物学系);and Cell Biology
Darius Catrina, Christian Bepler, Samuel Sledzieski, Rohit Singh
Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.
Rethinking Reverse KL as Adaptive Entropy Distillation
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Science and Technology, Hong Kong Metropolitan University(香港都会大学科技学院)
Entropy-Aware On-Policy Distillation of Language Models
IBM Research, San Jose, CA, USA(IBM研究院,旧金山,加州,美国);University of Toronto, Ontario, Canada(多伦多大学,安大略,加拿大);Vector Institute, Ontario, Canada(向量研究所,安大略,加拿大)
Department of Computer Science, University of California, Los Angeles, CA, USA(加州大学洛杉矶分校计算机科学系);Princeton AI Laboratory, Princeton University, Princeton, NJ, USA(普林斯顿大学普林斯顿AI实验室)
Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University(清华大学计算机科学与技术系、北京信息科学与技术国家研究中心、清华-博世机器学习中心);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Electronic Science and Technology of China(电子科技大学);South China University of Technology(华南理工大学)
本文提出Diffusion As Priors方法,通过量化合成与真实数据在特征空间的相似性提升数据集蒸馏的代表性,无需重新训练即可生成高质量数据集。
Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
Yandex(俄罗斯搜索引擎公司);HSE University(高等经济大学);University of Cambridge(剑桥大学)
机器学习模型的集合产生改进的系统性能以及鲁棒的和可解释的不确定性估计;然而,它们的推理成本通常可能过高。\emph{Ensemble Distribution Distillation}是一种允许单个模型有效地捕获集合的预测性能和不确定性估计的方法。对于分类,这是通过最大似然标准在集合成员的输出分布上训练狄利克雷分布来实现的。虽然理论上是原则性的,但当应用于类的数量非常高的大规模任务时,该标准表现出较差的收敛性。在我们的工作中,我们分析了这种效应,并表明低概率的狄利克雷对数似然准则类比高概率类产生更大的梯度。这迫使模型关注集合尾类概率的分布。我们提出了一个新的训练目标,最大限度地减少反向KL发散到一个\emph{Proxy-Dirichlet}目标来自合奏。这种损失解决了Encampaign Distribution Distillation的梯度问题,正如我们在ImageNet和WMT 17 En-De数据集上分别包含1000和40,000个类的理论和经验证明的那样。
Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
Hong Kong Baptist University(香港浸会大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);Ant Group(蚂蚁集团)
OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
The Hong Kong Polytechnic University(香港理工大学);Sun Yat-sen University(中山大学);PolyU-Daya Bay Technology and Innovation Research Institute(香港理工大学大亚湾技术创新研究院)