预训练分子语言模型与随机官能团掩码
Pre-trained Molecular Language Models with Random Functional Group Masking
AI总结:
本文提出一种基于SMILES的分子语言模型,通过随机掩码官能团对应的SMILES子序列来融入结构信息,在11个化学基准任务中9个超越现有预训练模型。
摘要:
计算化学领域的最新进展利用了基于Transformer的语言模型(如MoLFormer)的力量,这些模型通过海量简化分子输入行输入系统(SMILES)序列进行预训练,以理解和预测分子性质与活性,这是药物发现和材料科学等领域的关键步骤。为了进一步提升性能,研究者引入了基于图的分子表示(如图神经网络GEM),将分子的拓扑结构、几何形状、二维甚至三维结构纳入预训练。尽管现有研究中的大多数分子图是从SMILES序列自动转换而来,但可以假设基于Transformer的语言模型或许能够从SMILES序列中隐式学习结构感知表示。在本文中,我们提出了\ours{}——一种基于SMILES的分子语言模型,通过在预训练阶段随机掩码对应于特定分子官能团的SMILES子序列,来融入原子的结构信息。该技术旨在迫使模型更好地推断分子结构和性质,从而增强其预测能力。在化学领域的11个基准分类和回归任务上的广泛实验评估证明了\ours{}的稳健性和优越性。我们的发现表明,\ours{}在11个下游任务中的9个上优于现有的基于SMILES或图的预训练模型,在其余任务中则紧随其后,排名第二。
英文摘要:
Recent advancements in computational chemistry have leveraged the power of trans-former-based language models, such as MoLFormer, pre-trained using a vast amount of simplified molecular-input line-entry system (SMILES) sequences, to understand and predict molecular properties and activities, a critical step in fields like drug discovery and materials science. To further improve performance, researchers have introduced graph neural networks with graph-based molecular representations, such as GEM, incorporating the topology, geometry, 2D or even 3D structures of molecules into pre-training. While most of molecular graphs in existing studies were automatically converted from SMILES sequences, it is to assume that transformer-based language models might be able to implicitly learn structure-aware representations from SMILES sequences. In this paper, we propose \ours{} -- a SMILES-based \underline{\em M}olecular \underline{\em L}anguage \underline{\em M}odel, which randomly masking SMILES subsequences corresponding to specific molecular \underline{\em F}unctional \underline{\em G}roups to incorporate structure information of atoms during the pre-training phase. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities. Extensive experimental evaluations across 11 benchmark classification and regression tasks in the chemical domain demonstrate the robustness and superiority of \ours{}. Our findings reveal that \ours{} outperforms existing pre-training models, either based on SMILES or graphs, in 9 out of the 11 downstream tasks, ranking as a close second in the remaining ones.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。