arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.29529cond-mat.dis-nncs.LGq-bio.BM
收藏

在蛋白质结构预测中训练大语言模型时,中间温度下的采样是最佳的

Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction

Department of Physics, via Celoria 16, 20133 Milano, Italy(米兰大学物理系);Department of Physics and INFN, via Celoria 16, 20133 Milano, Italy(米兰大学物理系与意大利国家核物理研究院)

L. Ghiringhelli, A. Zambon, G. Tiana

AI总结:

研究通过统计力学框架分析transformer模型在蛋白质序列数据上的参数空间,发现transformer在蛋白质结构预测中表现优异的原因是其损失函数中缺乏一阶相变,从而在中间温度下具有良好的学习特性。

摘要:

我们通过统计力学框架研究了在蛋白质序列数据上训练的transformer模型的参数空间,通过兰格-动态在不同温度下采样损失景观,以表征低损失流形并理解transformer在蛋白质结构预测中表现优异的机制。我们发现,与前馈网络不同,transformer的损失中缺乏一阶相变,导致在某些中间温度下具有良好的学习特性。我们证明,如果嵌入维度是优化的,大多数层的参数在这些温度下高度保守,并提供了一种操作方法来找到此维度。最后,我们展示出在更高温度和更高嵌入维度下,注意力矩阵对蛋白质接触图的预测性优于那些用于学习的最佳维度。

英文摘要:

Using a statistical mechanics framework, we investigate the parameter space of transformer models trained on protein sequence data. We sample the loss landscape at varying temperatures using Langevin dynamics to characterize the low-loss manifold, and to understand the mechanisms underlying transformers' superior performance in protein structure prediction. We find that, at variance with networks not based on the attention mechanism, the lack of a first--order--like transition in the loss of the transformer produces a range of intermediate temperatures with good learning properties; this is true both for synthetic and natural protein sequences. We also show that the parameters of most layers are highly conserved at these temperatures if the dimension of the embedding is optimal, and we provide an operative way to find this dimension. Additionally, we show that the attention matrix is more predictive of the contact maps of the protein at higher temperatures and for higher dimensions of the embedding than those optimal for learning. Finally, we showed that the models sampled at intermediate temperatures can predict the free-energy variation upon mutation, better than models obtained through standard optimization techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑