Large pretrained models such as GPT-3 have had tremendous impact on modern natural language processing by leveraging self-supervised learning to learn salient representations that can be used to readily finetune on a wide variety of downstream tasks. We investigate the possibility of transferring such advances to molecular machine learning by building a chemical foundation model, ChemBERTa-2, using the language of SMILES. While labeled data for molecular prediction tasks is typically scarce, libraries of SMILES strings are readily available. In this work, we build upon ChemBERTa by optimizing the pretraining process. We compare multi-task and self-supervised pretraining by varying hyperparameters and pretraining dataset size, up to 77M compounds from PubChem. To our knowledge, the 77M set constitutes one of the largest datasets used for molecular pretraining to date. We find that with these pretraining improvements, we are competitive with existing state-of-the-art architectures on the MoleculeNet benchmark suite. We analyze the degree to which improvements in pretraining translate to improvement on downstream tasks.
ChemCLIP: Bridging Organic and Inorganic Anticancer Compounds Through Contrastive Learning
Li Ka Shing Faculty of Medicine, The University of Hong Kong(香港大学李嘉诚医学院);Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong(香港大学李嘉诚医学院诊断放射学系);Department of Chemistry, The University of Hong Kong(香港大学化学系)
PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design
Department of Applied Computer Science, The University of Winnipeg(应用计算机科学系,温尼伯大学);Department of Mechanical Engineering, University of Manitoba(机械工程系,曼尼托巴大学)
SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Prediction
Rensselaer Polytechnic Institute(伦斯勒理工学院);Stanford University(斯坦福大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Washington(华盛顿大学);Virginia Tech(弗吉尼亚理工大学)
MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules
Department of Chemical Engineering, Norwegian University of Science and Technology (NTNU)(挪威科学与技术大学化学工程系);Faculty of Industrial Engineering, KU Leuven(鲁文大学工业工程学院);University of Surrey(萨里大学)
Uni-Mol3: A Multi-Molecular Foundation Model for Advancing Organic Reaction Modeling
AI for Science Institute(人工智能科学研究院);DP Technology(DP技术);College of Chemistry and Molecular Engineering(化学与分子工程学院);School of Mathematical Sciences(数学科学学院);Center for Machine Learning Research(机器学习研究中心)
Drug-Target Interaction Prediction via Hierarchical Sequential Cross-Attention over Chemical and Protein Language Models
I2A institute, TÉLUQ University(I2A研究所,TÉLUQ大学);LIO, CRCHUM(LIO,蒙特利尔大学医院研究中心);LCSI, ESI(LCSI,阿尔及尔国立高等信息学院);CoMeT UR 24344, Universite de Poitiers(CoMeT UR 24344,普瓦捷大学);University of Ottawa(渥太华大学);Lebanese American University(黎巴嫩美国大学)
A Foundation Chemical Language Model for Comprehensive Fragment-Based Drug Discovery
Advanced Technology Core for Macromolecular X-Ray Crystallography(宏分子X射线衍射先进技术核心);Verna and Marrs McLean Dept. of Biochemistry and Molecular Pharmacology(韦纳和玛里斯麦克莱恩生物化学与分子药理学系);Dept. of Molecular & Cellular Biology(分子与细胞生物学系);Dept. of Molecular Virology & Microbiology(分子病毒学与微生物学系);Baylor College of Medicine(贝勒医学院)
Towards Foundation Model for Chemical Reactor Modeling: Meta-Learning with Physics-Informed Adaptation
Department of Chemical and Biomolecular Engineering, National University of Singapore(化学与生物分子工程系,新加坡国立大学);Artificial Intelligence Institute, National University of Singapore(人工智能研究所,新加坡国立大学)
Foundation Models for Discovery and Exploration in Chemical Space
Department of Mechanical Engineering, University of Michigan(密歇根大学机械工程系);Department of Chemical Engineering, University of Michigan(密歇根大学化学工程系);Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校电子与计算机工程系);The Santa Fe Institute(圣菲研究所);International Computer Science Institute(国际计算机科学研究所);Department of Statistics, University of California, Berkeley(加州大学伯克利分校统计学系);Department of Computer Science, University of Chicago(芝加哥大学计算机科学系);Argonne National Laboratory(阿贡国家实验室);Idaho National Laboratory(爱达荷国家实验室);NVIDIA Corporation(英伟达公司);Osmo Labs, PBC;AI Innovation Institute, Stony Brook University(石溪大学AI创新研究所);Brookhaven National Laboratory(布鲁赫斯研究所);Deep Forest Sciences, Palo Alto, CA(帕洛阿尔托的Deep Forest Sciences);Department of Aerospace Engineering, University of Michigan(密歇根大学航空航天工程系);Lawrence Berkeley National Laboratory(伯克利劳伦斯国家实验室)
MolSculpt: Sculpting 3D Molecular Geometries from Chemical Syntax
AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China(人工智能科学(AI4S)优选计划,北京大学深圳研究生院,中国);Faculty of Materials Science, Shenzhen MSU-BIT University, Shenzhen, China(材料科学学院,深圳MSU-BIT大学,中国)
Generative Chemical Language Models for Energetic Materials Discovery
Theoretical Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室理论部);Weapon Stockpile Modernization Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室武器库存现代化部);Computing and Artificial Intelligence Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室计算与人工智能部)
CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction
University of Toronto, Canada(多伦多大学);Vector Institute for Artificial Intelligence, Canada(人工智能向量研究所);Clean Energy Innovation Research Center, National Research Council, Canada(国家研究理事会清洁能源创新研究中心)
MicroDiffuse3D: A Foundation Model for 3D Microscopy Imaging Restoration
Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA(保罗·G·艾伦计算机科学与工程学院,华盛顿大学,西雅图,华盛顿州,美国);Department of Chemistry, University of Washington, Seattle, WA, USA(化学系,华盛顿大学,西雅图,华盛顿州,美国)
Tabular foundation models for in-context prediction of molecular properties
RWTH Aachen University, Process Systems Engineering (AVT.SVT)(RWTH亚琛大学,过程系统工程(AVT.SVT));Lehrstuhl Informatik 7, RWTH Aachen University(信息学7系,RWTH亚琛大学);Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems ICE-1: Energy Systems Engineering(焦耳研究中心有限公司,气候和能源系统ICE-1:能源系统工程);JARA-CSD
Can Molecular Foundation Models Know What They Don't Know? A Simple Remedy with Preference Optimization
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);Technical University of Berlin(柏林工业大学);University of Southern California(南加州大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所)