arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05797q-bio.BMcs.AIcs.CEcs.CLcs.LG
收藏

3D-MolT5:利用离散结构信息进行分子-文本建模

3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);Huazhong University of Science and Technology(华中科技大学);University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室)

Qizhi Pei, Rui Yan, Kaiyuan Gao, Jinhua Zhu, Lijun Wu

AI总结:

本文提出3D-MolT5框架,通过将细粒度3D分子子结构离散为专用token并联合多任务预训练,增强分子、结构与文本的跨模态对齐,在多项下游任务上优于现有方法。

摘要:

分子表示与自然语言表示的融合已成为分子科学中的一个焦点,而语言模型(LMs)的最新进展展现出对这两个领域进行综合建模的巨大潜力。然而,现有方法存在明显局限,尤其是忽视了三维(3D)信息,而这种信息对于理解分子结构和功能至关重要。尽管已有一些工作尝试利用外部结构编码模块将3D分子信息纳入LMs,但仍存在显著困难,例如预训练过程中跨模态交互不足以及模态对齐方面的挑战。为解决这些局限,我们提出3D-MolT5,这是一个旨在在序列空间和3D结构空间中同时建模分子的统一框架。我们方法的关键创新在于将细粒度3D子结构表示映射到专门的3D token词表中。该方法促进了序列与结构表示以token化格式无缝整合,使3D-MolT5能够在统一架构内编码分子序列、分子结构和文本序列。借助这种token化输入策略,我们构建了一个统一序列与结构数据格式的基础模型。随后,我们采用多任务目标进行联合预训练,以增强模型在共享表示空间中对这些多样模态的理解。因此,我们的方法显著改善了跨模态交互与对齐,解决了以往工作中的关键挑战。进一步的指令微调表明,3D-MolT5具有很强的泛化能力,并在多个下游任务中以更优性能超越现有方法。我们的代码可在https://github.com/QizhiPei/3D-MolT5获取。

英文摘要:

The integration of molecular and natural language representations has emerged as a focal point in molecular science, with recent advancements in Language Models (LMs) demonstrating significant potential for comprehensive modeling of both domains. However, existing approaches face notable limitations, particularly in their neglect of three-dimensional (3D) information, which is crucial for understanding molecular structures and functions. While some efforts have been made to incorporate 3D molecular information into LMs using external structure encoding modules, significant difficulties remain, such as insufficient interaction across modalities in pre-training and challenges in modality alignment. To address the limitations, we propose \textbf{3D-MolT5}, a unified framework designed to model molecule in both sequence and 3D structure spaces. The key innovation of our approach lies in mapping fine-grained 3D substructure representations into a specialized 3D token vocabulary. This methodology facilitates the seamless integration of sequence and structure representations in a tokenized format, enabling 3D-MolT5 to encode molecular sequences, molecular structures, and text sequences within a unified architecture. Leveraging this tokenized input strategy, we build a foundation model that unifies the sequence and structure data formats. We then conduct joint pre-training with multi-task objectives to enhance the model's comprehension of these diverse modalities within a shared representation space. Thus, our approach significantly improves cross-modal interaction and alignment, addressing key challenges in previous work. Further instruction tuning demonstrated that our 3D-MolT5 has strong generalization ability and surpasses existing methods with superior performance in multiple downstream tasks. Our code is available at https://github.com/QizhiPei/3D-MolT5.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑