arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11618cs.LGq-bio.QM
收藏

大规模化学语言模型在下游任务中的转移效果如何?

How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?

Graduate School of Pharmaceutical Sciences, Kyoto University(京都大学药学研究科);RIKEN BDR(理化学研究所BDR);Graduate School of Medicine, Kyoto University(京都大学医学研究科)

Tatsuya Sagawa, Ryosuke Kojima

AI总结:

研究评估了大规模化学语言模型在增加训练资源时对下游任务性能的影响,发现预训练损失降低但下游任务表现提升有限,并揭示了预训练评估与下游性能之间的差距。

摘要:

化学语言模型(CLMs)在大规模分子数据上预训练后广泛用于分子性质预测。然而,增加训练资源如模型大小、数据集大小和训练计算量能提高预训练损失和下游任务性能的常见信念在化学领域尚未系统验证。本文通过扩展训练资源预训练CLMs并测量跨多样分子性质预测(MPP)任务的转移性能来评估这一假设。我们发现,尽管预训练损失随着训练资源的增加而持续降低,下游任务性能却表现出有限的提升。此外,基于Hessian或损失景观的替代指标也未能估计CLMs的下游性能。我们进一步识别了即使预训练指标持续改进,下游性能饱和或退化的条件,并通过参数空间可视化分析了任务依赖的失败模式。这些结果揭示了预训练基于评估与下游性能之间的差距,并强调了需要考虑下游任务特征的模型选择和评估策略的重要性。

英文摘要:

Chemical Language Models (CLMs) pre-trained on large scale molecular data are widely used for molecular property prediction. However, the common belief that increasing training resources such as model size, dataset size, and training compute improves both pretraining loss and downstream task performance has not been systematically validated in the chemical domain. In this work, we evaluate this assumption by pretraining CLMs while scaling training resources and measuring transfer performance across diverse molecular property prediction (MPP) tasks. We find that while pretraining loss consistently decreases with increased training resources, downstream task performance shows limited improvement. Moreover, alternative metrics based on the Hessian or loss landscape also fail to estimate downstream performance in CLMs. We further identify conditions under which downstream performance saturates or degrades despite continued improvements in pretraining metrics, and analyze the underlying task dependent failure modes through parameter space visualizations. These results expose a gap between pretraining based evaluation and downstream performance, and emphasize the need for model selection and evaluation strategies that explicitly account for downstream task characteristics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑