We evaluate the impact of pretraining Graph Transformer architectures on atom-level quantum-mechanical features for the modeling of absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of drug-like compounds. We compare this pretraining strategy with two others: one based on molecular quantum properties (specifically the HOMO-LUMO gap) and one using a self-supervised atom masking technique. After fine-tuning on Therapeutic Data Commons ADMET datasets, we evaluate the performance improvement in the different models observing that models pretrained with atomic quantum mechanical properties produce in general better results. We then analyse the latent representations and observe that the supervised strategies preserve the pretraining information after finetuning and that different pretrainings produce different trends in latent expressivity across layers. Furthermore, we find that models pretrained on atomic quantum mechanical properties capture more low-frequency laplacian eigenmodes of the input graph via the attention weights and produce better representations of atomic environments within the molecule. Application of the analysis to a much larger non-public dataset for microsomal clearance illustrates generalizability of the studied indicators. In this case the performances of the models are in accordance with the representation analysis and highlight, especially for the case of masking pretraining and atom-level quantum property pretraining, how model types with similar performance on public benchmarks can have different performances on large scale pharmaceutical data.
The Absorption, Distribution, Metabolism, Elimination, and Toxicity (ADMET) properties of drug candidates are estimated to account for up to 50% of all clinical trial failures. Predicting ADMET properties has therefore been of great interest to the cheminformatics and medicinal chemistry communities in recent decades. Traditional cheminformatics approaches, whether the learner is a random forest or a deep neural network, leverage fixed fingerprint feature representations of molecules. In contrast, in this paper, we learn the features most relevant to each chemical task at hand by representing …
Google Cloud AI Research（谷歌云人工智能研究院）；Google Research（谷歌研究院）
Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed. However, we find the benefit brought by self-supervised pretraining on small molecular data can be negligible in many cases. We conduct thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset …
SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Prediction
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Stanford University（斯坦福大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Washington（华盛顿大学）；Virginia Tech（弗吉尼亚理工大学）
We propose the Graph Context Encoder (GCE), a simple but efficient approach for graph representation learning based on graph feature masking and reconstruction. GCE models are trained to efficiently reconstruct input graphs similarly to a graph autoencoder where node and edge labels are masked. In particular, our model is also allowed to change graph structures by masking and reconstructing graphs augmented by random pseudo-edges. We show that GCE can be used for novel graph generation, with applications for molecule generation. Used as a pretraining method, we also show that GCE improves base…
Department of Mechanical Engineering（机械工程系）；Center for Machine Intelligence and Data Science (CMInDS)（机器智能与数据科学中心）
High-energy materials (HEMs) are critical for propulsion and defense domains, yet their discovery remains constrained by experimental data and restricted access to testing facilities. This work presents a novel approach toward high-energy molecules by combining Long Short-Term Memory (LSTM) networks for molecular generation and Attentive Graph Neural Networks (GNN) for property predictions. We propose a transformative embedding space construction strategy that integrates fixed SHA-256 embeddings with partially trainable representations. Unlike conventional regularization techniques, this chang…
University College London（伦敦大学学院）；Alan Turing Institute（艾伦·图灵研究所）
Graph neural networks have become very popular for machine learning on molecules due to the expressive power of their learnt representations. However, molecular machine learning is a classically low-data regime and it isn't clear that graph neural networks can avoid overfitting in low-resource settings. In contrast, fingerprint methods are the traditional standard for low-data environments due to their reduced number of parameters and manually engineered features. In this work, we investigate whether graph neural networks are competitive in small data settings compared to the parametrically 'c…
Can we inject the pocket-ligand interaction knowledge into the pre-trained model and jointly learn their chemical space? Pretraining molecules and proteins has attracted considerable attention in recent years, while most of these approaches focus on learning one of the chemical spaces and lack the injection of biological knowledge. We propose a co-supervised pretraining (CoSP) framework to simultaneously learn 3D pocket and ligand representations. We use a gated geometric message passing layer to model both 3D pockets and ligands, where each node's chemical features, geometric position and ori…
Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets, followed by fine-tuning on smaller, labeled datasets. In this paper, we report MolE, a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs together with a …
Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion
Huawei Technologies Co., Ltd.（华为技术有限公司）；Chinese Academy of Sciences（中国科学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；Université de Montréal（蒙特利尔大学）
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；National Research Council Canada（加拿大国家研究委员会）；University of Ottawa（渥太华大学）；HEC Montréal（蒙特利尔高等商学院）
Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space fo…
International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Hong Kong University of Science and Technology（香港科技大学）；Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）
MOL-Mamba: Enhancing Molecular Representation with Structural & Electronic Insights
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Anhui University（安徽大学）；University of Science and Technology of China（中国科学技术大学）
Due to the nature of deep learning approaches, it is inherently difficult to understand which aspects of a molecular graph drive the predictions of the network. As a mitigation strategy, we constrain certain weights in a multi-task graph convolutional neural network according to the Gini index to maximize the "inequality" of the learned representations. We show that this constraint does not degrade evaluation metrics for some targets, and allows us to combine the outputs of the graph convolutional operation in a visually interpretable way. We then perform a proof-of-concept experiment on quant…
An accurate nucleic acid-small molecule docking framework via geometric deep learning with large-scale pretraining
College of Pharmaceutical Sciences, Zhejiang University（浙江大学药学院）；Faculty of Health Sciences, University of Macau（澳门大学健康科学学院）；Zhejiang Provincial Key Laboratory for Intelligent Drug Discovery and Development, Jinhua Institute of Zhejiang University（浙江省智能药物发现与开发重点实验室，浙江大学金华研究院）；Shanghai Innovation Institute（上海创新研究院）
Valence Labs（瓦伦斯实验室）；Université de Montréal（蒙特利尔大学）；Mila Quebec（米拉魁北克研究所）；McGill University（麦吉尔大学）；NVIDIA Corporation（英伟达公司）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
University of Toronto（多伦多大学）；Reverie Labs；DeepChem
GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well wi…
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
School of Mathematical Sciences（数学科学学院）；University of Electronic Science and Technology of China（电子科技大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Oakland University（奥克兰大学）
Multimodal Molecular Pretraining via Modality Blending
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Harbin Institute of Technology（哈尔滨工业大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
MS-BART: Unified Modeling of Mass Spectra and Molecules for Structure Elucidation
X-LANCE Lab, School of Computer Science MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China（X-LANCE实验室，计算机科学学院，人工智能教育部重点实验室，上海交通大学人工智能研究院，上海，中国）；Suzhou Laboratory, Suzhou, China（苏州实验室，苏州，中国）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院，上海，中国）；Jiangsu Key Lab of Language Computing, Suzhou, China（江苏省语言计算重点实验室，苏州，中国）
An Equivariant Pretrained Transformer for Unified 3D Molecular Representation Learning
Tsinghua University（清华大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Renmin University of China（中国人民大学）；University of Pisa（比萨大学）
Cyclica Inc.（Cyclica公司）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
The prediction of absorption, distribution, metabolism, excretion, and toxicity (ADMET) of small molecules from their molecular structure is a central problem in medicinal chemistry with great practical importance in drug discovery. Creating predictive models conventionally requires substantial trial-and-error for the selection of molecular representations, machine learning (ML) algorithms, and hyperparameter tuning. A generally applicable method that performs well on all datasets without tuning would be of great value but is currently lacking. Here, we describe Pareto-Optimal Embedded Modelin…
Machine learning approaches have become popular for molecular modeling tasks, including molecular force fields and properties prediction. Traditional supervised learning methods suffer from scarcity of labeled data for particular tasks, motivating the use of large-scale dataset for other relevant tasks. We propose to pretrain neural networks on a dataset of 86 millions of molecules with atom charges and 3D geometries as inputs and molecular energies as labels. Experiments show that, compared to training from scratch, fine-tuning the pretrained model can significantly improve the performance fo…
Vancouver Prostate Centre（温哥华前列腺中心）；The University of British Columbia（不列颠哥伦比亚大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Recursion Pharmaceuticals（递归制药公司）
Ardigen（阿迪根公司）；Jagiellonian University（雅盖隆大学）；New York University（纽约大学）
Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple se…
