In recent years, there has been a surge in the development of 3D structure-based pre-trained protein models, representing a significant advancement over pre-trained protein language models in various downstream tasks. However, most existing structure-based pre-trained models primarily focus on the residue level, i.e., alpha carbon atoms, while ignoring other atoms like side chain atoms. We argue that modeling proteins at both residue and atom levels is important since the side chain atoms can also be crucial for numerous downstream tasks, for example, molecular docking. Nevertheless, we find that naively combining residue and atom information during pre-training typically fails. We identify a key reason is the information leakage caused by the inclusion of atom structure in the input, which renders residue-level pre-training tasks trivial and results in insufficiently expressive residue representations. To address this issue, we introduce a span mask pre-training strategy on 3D protein chains to learn meaningful representations of both residues and atoms. This leads to a simple yet effective approach to learning protein representation suitable for diverse downstream tasks. Extensive experimental results on binding site prediction and function prediction tasks demonstrate our proposed pre-training approach significantly outperforms other methods. Our code will be made public.
Mask prior-guided denoising diffusion improves inverse protein folding
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Biologics Engineering, Oncology R&D, AstraZeneca（阿斯利康生物制药工程与肿瘤研发部）；Medicinal Chemistry, Research and Early Development, Cardiovascular, Renal and Metabolism, BioPharmaceuticals R&D, AstraZeneca（阿斯利康医药化学与研发部）；School of Biological Sciences, University of Southampton（南安普顿大学生物科学学院）；Centre for Machine Intelligence, University of Sheffield（谢菲尔德大学智能中心）
University of Washington（华盛顿大学）；Salesforce Research（赛富时研究院）
For protein sequence datasets, unlabeled data has greatly outpaced labeled data due to the high cost of wet-lab characterization. Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. However, the optimal pre-training strategy remains an open question. Instead of strictly borrowing from natural language processing (NLP) in the form of masked or autoregressive language modeling, we introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments.…
Joint Masked Reconstruction and Contrastive Learning for Mining Interactions Between Proteins
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；Institute of Artificial Intelligence, Huazhong University of Science and Technology（华中科技大学人工智能研究院）；Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology（华中科技大学湖北省类脑智能系统重点实验室）；Key Laboratory of Image Processing and Intelligent Control (Huazhong University of Science and Technology), Ministry of Education（华中科技大学图像信息处理与智能控制教育部重点实验室）
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；OpenDataLab, Shanghai Artificial Intelligence Laboratory（上海人工智能实验室 OpenDataLab）；Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；East China Normal University（华东师范大学）；Zhongguancun Academy（中关村学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Masking tokens uniformly at random constitutes a common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM to minimize its training objective by latching onto shallow local signals, leading to pretraining inefficiency and suboptimal downstream performance. To address this flaw, we propose PMI-Masking, a principled masking strategy based on the concept of Pointwise Mutual Information (PMI), which jointly masks a token n-gram if it exhibits high collocation over the corpus. PMI-Masking motivates, unifies, and improves upon prior…
The University of Hong Kong（香港大学）；Xiaohe Healthcare, ByteDance（字节跳动小荷健康）；JancsiTech（Jancsi科技）；OPPO HealthLab（OPPO健康实验室）
Protein representation learning has primarily benefited from the remarkable development of language models (LMs). Accordingly, pre-trained protein models also suffer from a problem in LMs: a lack of factual knowledge. The recent solution models the relationships between protein and associated knowledge terms as the knowledge encoding objective. However, it fails to explore the relationships at a more granular level, i.e., the token level. To mitigate this, we propose Knowledge-exploited Auto-encoder for Protein (KeAP), which performs token-level knowledge graph exploration for protein represen…
GLProtein: Global-and-Local Structure Aware Protein Representation Learning
Department of Computing (COMP), The Hong Kong Polytechnic University（计算机学院（COMP），香港理工大学）；Department of Management and Marketing (MM), The Hong Kong Polytechnic University（管理与营销学院（MM），香港理工大学）；Department of Computer Science, Sichuan University（计算机学院，四川大学）
Extracting Inter-Protein Interactions Via Multitasking Graph Structure Learning
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；College of Life Sciences, South-Central Minzu University（中南民族大学生命科学学院）
Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training
Harvard Medical School（哈佛医学院）；University of Copenhagen（哥本哈根大学）；Novo Nordisk A/S（诺和诺德公司）；Microsoft Research（微软研究院）；Technical University of Denmark（丹麦技术大学）
3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health（亥姆霍兹慕尼黑中心 - 德国环境健康研究中心，计算健康中心，健康人工智能研究所与亥姆霍兹人工智能）；Department of Medicine III, Ludwig-Maximilian-University Hospital（路德维希-马克西米利安大学医院第三内科）；Department of Physics, Ludwig-Maximilian-University（路德维希-马克西米利安大学物理系）；German Cancer Consortium (DKTK), partner site Munich（德国癌症联盟（DKTK）慕尼黑合作站点）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
Amazon Devices（亚马逊设备部门）；Amazon Machine Learning Solutions Lab（亚马逊机器学习解决方案实验室）；AWS Professional Services（亚马逊云科技专业服务部门）
Representation learning for proteins has primarily focused on the global understanding of protein sequences regardless of their length. However, shorter proteins (known as peptides) take on distinct structures and functions compared to their longer counterparts. Unfortunately, there are not as many naturally occurring peptides available to be sequenced and therefore less peptide-specific data to train with. In this paper, we propose a new peptide data augmentation scheme, where we train peptide language models on artificially constructed peptides that are small contiguous subsets of longer, wi…
Generative power of a protein language model trained on multiple sequence alignments
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Institute of Bioengineering（生物工程研究所）；School of Life Sciences（生命科学学院）；SIB Swiss Institute of Bioinformatics（瑞士生物信息学研究所）
Have you ever been troubled by the complexity and computational cost of SE(3) protein structure modeling and been amazed by the simplicity and power of language modeling? Recent work has shown promise in simplifying protein structures as sequences of protein angles; therefore, language models could be used for unconstrained protein backbone generation. Unfortunately, such simplification is unsuitable for the constrained protein inpainting problem, where the model needs to recover masked structures conditioned on unmasked ones, as it dramatically increases the computing cost of geometric constr…
Despite being self-supervised, protein language models have shown remarkable performance in fundamental biological tasks such as predicting impact of genetic variation on protein structure and function. The effectiveness of these models on diverse set of tasks suggests that they learn meaningful representations of fitness landscape that can be useful for downstream clinical applications. Here, we interrogate the use of these language models in characterizing known pathogenic mutations in curated, medically actionable genes through an exhaustive search of putative compensatory mutations on each…
Facebook AI Research（Facebook人工智能研究院）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；Allen Institute of Artificial Intelligence（艾伦人工智能研究所）
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size a…
DISPROTBENCH: Uncovering the Functional Limits of Protein Structure Prediction Models in Intrinsically Disordered Regions
Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎-罗克琴堡研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒研究实验室）
Seoul National University（首尔国立大学）；LG AI Research（LG人工智能研究院）；NAVER Corp.（NAVER公司）；Kangwon National University（江原国立大学）；Seoul National University of Science and Technology（首尔科技大学）
Bridging the exponentially growing gap between the numbers of unlabeled and labeled protein sequences, several studies adopted semi-supervised learning for protein sequence modeling. In these studies, models were pre-trained with a substantial amount of unlabeled data, and the representations were transferred to various downstream tasks. Most pre-training methods solely rely on language modeling and often exhibit limited performance. In this paper, we introduce a novel pre-training scheme called PLUS, which stands for Protein sequence representations Learned Using Structural information. PLUS …
Structure Language Models for Protein Conformation Generation
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；University of Ottawa（渥太华大学）；National Research Council Canada（加拿大国家研究委员会）；CIFAR AI Chair（加拿大高级研究所人工智能主席项目）；HEC Montréal（蒙特利尔高等商学院）
Experimental determination of protein function is resource-consuming. As an alternative, computational prediction of protein function has received attention. In this context, protein structural classification (PSC) can help, by allowing for determining structural classes of currently unclassified proteins based on their features, and then relying on the fact that proteins with similar structures have similar functions. Existing PSC approaches rely on sequence-based or direct 3-dimensional (3D) structure-based protein features. In contrast, we first model 3D structures of proteins as protein st…
Technical University of Munich (TUM)（慕尼黑工业大学）；Proteinea, Inc.（Proteinea公司）；Columbia University（哥伦比亚大学）
As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the f…
Traditional machine learning vs. deep learning from dynamic graph representations of proteins' 3D folds in the task of protein structure classification
