arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.01712cs.LGcs.AIq-bio.BM
收藏

ChemBERTa-2:迈向化学基础模型

ChemBERTa-2: Towards Chemical Foundation Models

Reverie Labs(遐想实验室);UC Berkeley(加州大学伯克利分校);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Deep Forest Sciences(深林科学公司)

Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

摘要:

GPT-3等大型预训练模型通过利用自监督学习来学习可用于轻松微调各种下游任务的突出表示,对现代自然语言处理产生了巨大影响。我们研究了将这些进展转移到分子机器学习的可能性,通过使用SMILES语言构建化学基础模型ChemBERTA-2。虽然用于分子预测任务的标记数据通常是稀缺的,但SMILES字符串的库是容易获得的。在这项工作中,我们通过优化预训练过程来构建ChemBERTA。我们通过不同的超参数和预训练数据集大小来比较多任务和自我监督预训练,PubChem的化合物多达77 M。据我们所知,77 M集构成了迄今为止用于分子预训练的最大数据集之一。我们发现,通过这些预训练改进,我们在MoleculeNet基准测试套件上与现有的最先进的架构竞争。我们分析了预训练的改进转化为下游任务改进的程度。

英文摘要:

Large pretrained models such as GPT-3 have had tremendous impact on modern natural language processing by leveraging self-supervised learning to learn salient representations that can be used to readily finetune on a wide variety of downstream tasks. We investigate the possibility of transferring such advances to molecular machine learning by building a chemical foundation model, ChemBERTa-2, using the language of SMILES. While labeled data for molecular prediction tasks is typically scarce, libraries of SMILES strings are readily available. In this work, we build upon ChemBERTa by optimizing the pretraining process. We compare multi-task and self-supervised pretraining by varying hyperparameters and pretraining dataset size, up to 77M compounds from PubChem. To our knowledge, the 77M set constitutes one of the largest datasets used for molecular pretraining to date. We find that with these pretraining improvements, we are competitive with existing state-of-the-art architectures on the MoleculeNet benchmark suite. We analyze the degree to which improvements in pretraining translate to improvement on downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑