arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.20267cs.LGcs.AIphysics.chem-ph
收藏

面向化学语言的大型编码器-解码器基础模型家族

A Large Encoder-Decoder Family of Foundation Models For Chemical Language

IBM Research Brazil(IBM巴西研究所以及IBM Research Brazil);IBM Research Almaden(IBM阿马登研究所)

Eduardo Soares, Victor Shirasuna, Emilio Vital Brazil, Renato Cerqueira, Dmitry Zubarev, Kristin Schmidt

AI总结:

本文提出在9100万SMILES样本上预训练的大型编码器-解码器化学基础模型,支持量子性质预测等多种任务,在多个基准上取得最优结果,并验证了嵌入空间的可分离性与少样本学习能力。

摘要:

化学语言模型的大规模预训练方法代表了化学信息学领域的一项突破。这些方法通过在大规模无标注语料库上进行自监督学习,学习输入标记的上下文表示,在性质预测和分子生成等任务中表现出色。通常,这一过程包括在无标注数据上进行预训练,然后在特定任务上进行微调,从而减少对标注数据集的依赖,并拓宽化学语言表示的理解能力。本文介绍了一个大型编码器-解码器化学基础模型,该模型在从PubChem获取的9100万个SMILES样本的精选数据集上进行预训练,该数据集相当于40亿个分子标记。所提出的基础模型支持不同的复杂任务,包括量子性质预测,并提供两个主要变体(289M和$8\ imes289M$)的灵活性。我们在多个基准数据集上的实验验证了所提模型在不同任务上提供最先进结果的能力。我们还对嵌入空间的组合性进行了初步评估,将其作为推理任务的先决条件。我们证明,与最先进的方法相比,所生成的潜在空间是可分离的,并具有少样本学习能力。

英文摘要:

Large-scale pre-training methodologies for chemical language models represent a breakthrough in cheminformatics. These methods excel in tasks such as property prediction and molecule generation by learning contextualized representations of input tokens through self-supervised learning on large unlabeled corpora. Typically, this involves pre-training on unlabeled data followed by fine-tuning on specific tasks, reducing dependence on annotated datasets and broadening chemical language representation understanding. This paper introduces a large encoder-decoder chemical foundation models pre-trained on a curated dataset of 91 million SMILES samples sourced from PubChem, which is equivalent to 4 billion of molecular tokens. The proposed foundation model supports different complex tasks, including quantum property prediction, and offer flexibility with two main variants (289M and $8\times289M$). Our experiments across multiple benchmark datasets validate the capacity of the proposed model in providing state-of-the-art results for different tasks. We also provide a preliminary assessment of the compositionality of the embedding space as a prerequisite for the reasoning tasks. We demonstrate that the produced latent space is separable compared to the state-of-the-art with few-shot learning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑