arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.04476cs.CLcs.AI
收藏

使用上下文感知分词训练文本到分子模型

Training Text-to-Molecule Models with Context-Aware Tokenization

Seoul National University(首尔国立大学);Moloco Inc.(Moloco公司);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院)

Seojin Kim, Hyeontae Song, Jaehyun Nam, Jinwoo Shin

AI总结:

提出文本到分子模型CAMT5,通过引入子结构级分词和基于重要性的训练策略,更好地捕捉分子全局结构上下文与语义,仅用2%训练token即超越SOTA,并辅以集成策略提升性能。

摘要:

最近,文本到分子模型在各种化学应用(例如药物发现)中展现出了巨大潜力。这些模型通过将分子表示为原子序列来使语言模型适应分子数据。然而,它们依赖于原子级分词,这主要侧重于建模局部连接性,从而限制了模型捕捉分子内全局结构上下文的能力。为了解决这个问题,我们提出了一种新颖的文本到分子模型,命名为 Context-Aware Molecular T5 (CAMT5)。受子结构级上下文(例如环系统)在理解分子结构中的重要性的启发,我们为文本到分子模型引入了子结构级分词。基于我们的分词方案,我们开发了一种基于重要性的训练策略,优先考虑关键子结构,使 CAMT5 能够更好地捕捉分子语义。大量实验验证了 CAMT5 在各种文本到分子生成任务中的优越性。有趣的是,我们发现 CAMT5 仅使用 2% 的训练 token 就优于了最先进的方法。此外,我们提出了一种简单而有效的集成策略,聚合文本到分子模型的输出以进一步提升生成性能。代码可在 https://github.com/Songhyeontae/CAMT5.git 获取。

英文摘要:

Recently, text-to-molecule models have shown great potential across various chemical applications, e.g., drug-discovery. These models adapt language models to molecular data by representing molecules as sequences of atoms. However, they rely on atom-level tokenizations, which primarily focus on modeling local connectivity, thereby limiting the ability of models to capture the global structural context within molecules. To tackle this issue, we propose a novel text-to-molecule model, coined Context-Aware Molecular T5 (CAMT5). Inspired by the significance of the substructure-level contexts in understanding molecule structures, e.g., ring systems, we introduce substructure-level tokenization for text-to-molecule models. Building on our tokenization scheme, we develop an importance-based training strategy that prioritizes key substructures, enabling CAMT5 to better capture the molecular semantics. Extensive experiments verify the superiority of CAMT5 in various text-to-molecule generation tasks. Intriguingly, we find that CAMT5 outperforms the state-of-the-art methods using only 2% of training tokens. In addition, we propose a simple yet effective ensemble strategy that aggregates the outputs of text-to-molecule models to further boost the generation performance. Code is available at https://github.com/Songhyeontae/CAMT5.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑