发表机构
University of Oxford; Adelaide University; Slater Labs（牛津大学; 阿德莱德大学; 斯莱特实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出程序化预训练三阶段流程，通过抽象程序化数据学习归纳偏置，在分子性质预测中显著提升性能，尤其在数据稀缺时效果最佳。
AI 中文摘要
分子性质预测常常受到下游标记数据集规模小的限制，这促使人们在大规模未标记分子语料库上进行预训练。在这项工作中，我们探讨是否可以在模型看到任何分子数据之前，从抽象的、程序化生成的数据中学习到有用的归纳偏置。我们引入了一个三阶段训练流程，包括程序化预训练、基于SMILES的分子预训练以及下游微调，并评估了涵盖序列结构、元胞自动机和图推理的多个程序化任务。我们发现，即使在后续的分子预训练之后，程序化预训练也能改善分子性质预测：在亲脂性任务上，\textsc{Reverse}将测试误差降低了4.8%。作为背景，这一改进幅度大约相当于我们25万分子基线与公开的、在约1亿分子上预训练的MoLFormer检查点之间性能差异的90%。我们的分析表明，这种益处在下游数据稀缺时最为显著，依赖于程序化数据的结构而不仅仅是表面统计特征，并且不会随着额外的程序化训练而单调增加。相反，迁移通常在中等程序化预算时达到峰值，并在模型接近程序化任务收敛时恶化。我们进一步发现，对于多个任务，大部分可迁移信息集中在注意力层，而前馈层可能导致过度特化。这些结果表明，程序化数据可以为分子学习提供可迁移的结构，并在标记分子数据有限时提供一条补充性的性能提升途径。
英文摘要
Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules. In this work, we ask whether useful inductive biases can instead be learned from abstract, procedurally generated data before a model sees any molecular data. We introduce a three-stage training pipeline consisting of procedural pretraining, molecular pretraining on SMILES, and downstream fine-tuning, and evaluate several procedural tasks spanning sequence structure, cellular automata, and graph reasoning. We find that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining: on Lipophilicity, \textsc{Reverse} reduces test error by 4.8\%. For context, the magnitude of this improvement is roughly 90\% of the performance difference between our 250K-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules. Our analysis shows that the benefit is strongest under downstream data scarcity, depends on the structure of the procedural data rather than only surface-level statistics, and does not increase monotonically with additional procedural training. Instead, transfer typically peaks at an intermediate procedural budget and deteriorates as the model approaches convergence on the procedural task. We further find that, for several tasks, much of the transferable information is localized in the attention layers, while feed-forward layers can contribute to over-specialization. These results show that procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited.
Comments14 pages
