arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.05540q-bio.QMcs.AIcs.CLcs.LG
收藏

面向蛋白质理解的大语言模型微调数据集与基准

A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding

Toursun Synbio(图森合成生物);Johns Hopkins University(约翰斯·霍普金斯大学);University of Cambridge(剑桥大学);Shanghai Institute for Biomedical and Pharmaceutical Technologies(上海生物医药技术研究院);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);UNSW Syndey(新南威尔士大学悉尼分校)

Yiqing Shen, Zan Chen, Michail Mamalakis, Luhan He, Haiyang Xia, Tianbin Li, Yanzhou Su, Junjun He, Yu Guang Wang

AI总结:

本文构建含174.6亿预训练token与89.3万SFT指令的ProteinLMDataset,以及含944道人工核验多选题的ProteinLMBench;经其训练的InternLM2-7B在蛋白质理解评测中超过GPT-4。

摘要:

蛋白质序列与自然语言在序列结构上的相似性,启发了将大语言模型(LLM)应用于蛋白质理解。尽管LLM在NLP中取得了成功,但它们理解蛋白质序列的有效性仍是一个悬而未决的问题,这在很大程度上是因为缺少将蛋白质序列与描述性文本联系起来的数据集。随后,研究人员尝试通过将蛋白质序列编码器与预训练LLM相结合,使LLM适配蛋白质理解。然而,这种适配提出了一个根本问题:“最初为NLP设计的LLM能否有效地将蛋白质序列理解为一种语言形式?”由于蛋白质序列与相应文本描述之间缺乏直接关联,现有数据集不足以解决这一问题,从而限制了有效训练和评估LLM蛋白质理解能力的空间。为弥补这一差距,我们推出了ProteinLMDataset,这是一个专门用于进一步对LLM进行自监督预训练和监督微调(SFT)的数据集,以增强其蛋白质序列理解能力。具体而言,ProteinLMDataset包含用于预训练的174.6亿个token,以及用于SFT的89.3万条指令。此外,我们提出了ProteinLMBench,这是首个由944道经人工核验的多项选择题组成的基准数据集,用于评估LLM的蛋白质理解能力。ProteinLMBench以多种语言纳入蛋白质相关细节和序列,为评估LLM的蛋白质理解能力建立了新标准。在ProteinLMDataset上进行预训练和微调的大语言模型InternLM2-7B在ProteinLMBench上优于GPT-4,取得了最高准确率。

英文摘要:

The parallels between protein sequences and natural language in their sequential structures have inspired the application of large language models (LLMs) to protein understanding. Despite the success of LLMs in NLP, their effectiveness in comprehending protein sequences remains an open question, largely due to the absence of datasets linking protein sequences to descriptive text. Researchers have then attempted to adapt LLMs for protein understanding by integrating a protein sequence encoder with a pre-trained LLM. However, this adaptation raises a fundamental question: "Can LLMs, originally designed for NLP, effectively comprehend protein sequences as a form of language?" Current datasets fall short in addressing this question due to the lack of a direct correlation between protein sequences and corresponding text descriptions, limiting the ability to train and evaluate LLMs for protein understanding effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset specifically designed for further self-supervised pretraining and supervised fine-tuning (SFT) of LLMs to enhance their capability for protein sequence comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for pretraining and 893,000 instructions for SFT. Additionally, we present ProteinLMBench, the first benchmark dataset consisting of 944 manually verified multiple-choice questions for assessing the protein understanding capabilities of LLMs. ProteinLMBench incorporates protein-related details and sequences in multiple languages, establishing a new standard for evaluating LLMs' abilities in protein comprehension. The large language model InternLM2-7B, pretrained and fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench, achieving the highest accuracy score.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑