蛋白质语言模型偏离自然语言：比较分析与改进推理
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
AI总结：
本研究通过比较蛋白质与自然语言领域中信息分布差异，改进蛋白质非结构性质预测的准确性和效率。
摘要：
现代蛋白质语言模型（PLMs）将自然语言处理中的变压器架构应用于生物序列，预测多种蛋白质功能和性质。然而，蛋白质语言与自然语言有关键差异，例如尽管只有20种氨基酸的词汇，但功能空间却十分丰富。这些差异促使研究探讨变压器架构在蛋白质领域中的运作差异以及如何更好地利用PLMs解决蛋白质相关任务。在本工作中，我们首先直接比较蛋白质和自然语言领域中信息在注意力头层分布的差异。此外，我们采用一种简单的早退技术——最初用于自然语言领域以在牺牲性能的情况下提高效率——通过允许模型自动选择特定任务和蛋白质的中间层表示来实现非结构性质预测的准确性和效率的双重提升。我们实现了从0.4到7.01个百分点的性能提升，同时在模型和非结构预测任务上将效率提高超过10%。我们的工作开辟了直接比较语言模型在进入蛋白质领域时行为变化的研究领域，并推动了生物领域的语言建模发展。
英文摘要：
Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
