Uni-Mol2：大规模探索分子预训练模型
Uni-Mol2: Exploring Molecular Pretraining Model at Scale
AI总结：
本文提出分子预训练模型Uni-Mol2，通过双轨transformer融合原子、图与几何特征，并系统揭示分子预训练中的缩放定律，最终扩展至11亿参数，在QM9和COMPAS-1D上显著超越现有方法。
摘要：
近年来，预训练模型在自然语言处理（NLP）、计算机视觉（CV）和生命科学领域取得了显著进展。NLP和CV领域的显著进步主要由模型参数和数据规模的扩张驱动，这一现象如今被公认为缩放定律。然而，探索分子预训练模型中缩放定律的研究仍属空白。在本工作中，我们提出了Uni-Mol2，一种创新的分子预训练模型，它利用双轨transformer有效整合原子级、图级和几何结构级特征。与此同时，我们系统地研究了分子预训练模型中的缩放定律，刻画了验证损失与模型规模、数据集规模和计算资源之间的幂律相关性。据此，我们通过在8亿个构象上预训练，成功将Uni-Mol2扩展到11亿参数，使其成为迄今为止最大的分子预训练模型。大量实验表明，随着模型规模增长，下游任务持续改善。具有11亿参数的Uni-Mol2也优于现有方法，在QM9数据集上平均提升27%，在COMPAS-1D数据集上平均提升14%。
英文摘要：
In recent years, pretraining models have made significant advancements in the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the expansion of model parameters and data size, a phenomenon now recognized as the scaling laws. However, research exploring scaling law in molecular pretraining models remains unexplored. In this work, we present Uni-Mol2 , an innovative molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show consistent improvement in the downstream tasks as the model size grows. The Uni-Mol2 with 1.1B parameters also outperforms existing methods, achieving an average 27% improvement on the QM9 and 14% on COMPAS-1D dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
