arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07710cs.LGq-bio.BM
收藏

反蒸馏:一致地扩展蛋白质语言模型表示

Reverse Distillation: Consistently Scaling Protein Language Model Representations

Department of Computer Science(计算机科学系);Duke University(杜克大学);Center for Computational Biology(计算生物学中心);Flatiron Institute(Flatiron研究所);Depts of Biostats. and Bioinformatics(生物统计学与生物信息学系;细胞生物学系);and Cell Biology

Darius Catrina, Christian Bepler, Samuel Sledzieski, Rohit Singh

AI总结:

反蒸馏通过分解大型蛋白质语言模型表示为正交子空间,使较大模型在相同维度下始终优于较小模型,提升了蛋白质特征编码效果。

摘要:

与自然语言处理和计算机视觉中可预测的扩展定律不同,蛋白质语言模型(PLMs)扩展效果较差:在许多任务中,同一家族中的模型性能在达到一定规模后趋于平缓甚至下降,中等规模的模型往往优于最大的模型。我们引入了反蒸馏,一种系统性的框架,该框架通过同一家族中的较小模型引导,将大型PLM表示分解为正交的子空间。所得到的嵌入具有嵌套的马特洛什卡式结构:较大模型嵌入的前k个维度正好是较小模型的表示。这确保了反蒸馏的较大模型始终优于较小的模型。一个推动性的直觉是,受容量限制的较小模型倾向于优先编码广泛共享的蛋白质特征。反蒸馏分离了这些共享特征,并从较大模型中正交提取额外的贡献,防止了两者之间的干扰。在ProteinGym基准测试中,反蒸馏的ESM-2变体在相同嵌入维度下优于各自的基础模型,其中反蒸馏的150亿参数模型取得了最强的性能。我们的框架可以推广到任何存在扩展挑战的模型家族。代码和训练好的模型可在https://github.com/rohitsinghlab/plm_reverse_distillation上获得。

英文摘要:

Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑