This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioner, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance. Code is released at \url{https://github.com/bytedance/dplm}.
Harvard University（哈佛大学）；Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学博德研究所）；Harvard Data Science（哈佛数据科学研究院）
Spatial context is central to understanding health and disease. Yet reference protein interaction networks lack such contextualization, thereby limiting the study of where protein interactions likely occur in the human body and how they may be altered in disease. Contextualized protein interactions could better characterize genes with disease-specific interactions and elucidate diseases' manifestation in specific cell types. Here, we introduce AWARE, a graph neural message passing approach to inject cellular and tissue context into protein embeddings. AWARE optimizes for a multi-scale embeddin…
University of Trieste（特里埃斯特大学）；Abdus Salam International Centre for Theoretical Physics (ICTP)（阿布杜斯·萨拉姆国际理论物理学中心（ICTP））；Scuola Internazionale Superiore di Studi Avanzati (SISSA)（国际先进研究高等学院（SISSA））；University of Nova Gorica（诺瓦戈里察大学）；INFN – National Institute for Nuclear Physics（意大利国家核物理研究所（INFN））；ICSC - Centro Nazionale di Ricerca in High Performance Computing（高性能计算国家研究中心（ICSC））；Imperial College London（伦敦帝国理工学院）
Lifelong Learner: Discovering Versatile Neural Solvers for Vehicle Routing Problems
department of Computer Science at National Yang Ming Chiao Tung University（国立阳明交通大学计算机科学系）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（资讯通信研究院，科技研究局（A*STAR））；College of Computing and Data Science at Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science, Sichuan Normal University（四川师范大学计算机科学学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Center for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究中心，科技研究局（A*STAR））
3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health（亥姆霍兹慕尼黑中心 - 德国环境健康研究中心，计算健康中心，健康人工智能研究所与亥姆霍兹人工智能）；Department of Medicine III, Ludwig-Maximilian-University Hospital（路德维希-马克西米利安大学医院第三内科）；Department of Physics, Ludwig-Maximilian-University（路德维希-马克西米利安大学物理系）；German Cancer Consortium (DKTK), partner site Munich（德国癌症联盟（DKTK）慕尼黑合作站点）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training
Beihang University（北京航空航天大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；State Grid Smart Grid Research Institute Co., Ltd.（国网智能电网研究院有限公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）；Massachusetts Institute of Technology（麻省理工学院）
Antibodies are versatile proteins that bind to pathogens like viruses and stimulate the adaptive immune system. The specificity of antibody binding is determined by complementarity-determining regions (CDRs) at the tips of these Y-shaped proteins. In this paper, we propose a generative model to automatically design the CDRs of antibodies with enhanced binding specificity or neutralization capabilities. Previous generative approaches formulate protein design as a structure-conditioned sequence generation task, assuming the desired 3D structure is given a priori. In contrast, we propose to co-de…
Customizing Spider Silk: Generative Models with Mechanical Property Conditioning for Protein Engineering
Division of Robotics, Perception and Learning（机器人、感知与学习 division）；KTH Royal Institute of Technology（皇家理工学院）；Department of Animal Biosciences（动物生物学系）；Swedish University of Agricultural Sciences (SLU)（瑞典农业科学大学）；Science for Life Laboratory (SciLifeLab)（生命科学实验室）；Uppsala University（乌普萨拉大学）
The Hong Kong University of Science and Technology（香港科技大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writing (image-to-text generation) and painting (text-to-image generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVin…
Computer Learning Research Centre, Royal Holloway, University of London（伦敦大学皇家霍洛威学院计算机学习研究中心）；School of Biological Sciences, Royal Holloway, University of London（伦敦大学皇家霍洛威学院生物科学学院）
Huazhong University of Science and Technology（华中科技大学）；Microsoft Research AI4Science（微软研究院AI4Science）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）
Antibodies are versatile proteins that can bind to pathogens and provide effective protection for human body. Recently, deep learning-based computational antibody design has attracted popular attention since it automatically mines the antibody patterns from data that could be complementary to human experiences. However, the computational methods heavily rely on high-quality antibody structure data, which is quite limited. Besides, the complementarity-determining region (CDR), which is the key component of an antibody that determines the specificity and binding affinity, is highly variable and …
Making Pre-trained Language Models Great on Tabular Prediction
The Second Affiliated Hospital Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Zhejiang University（浙江大学）；University of Notre Dame（圣母大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
