For protein sequence datasets, unlabeled data has greatly outpaced labeled data due to the high cost of wet-lab characterization. Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. However, the optimal pre-training strategy remains an open question. Instead of strictly borrowing from natural language processing (NLP) in the form of masked or autoregressive language modeling, we introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments. Using a set of five, standardized downstream tasks for protein models, we demonstrate that our pre-training task along with a multi-task objective outperforms masked language modeling alone on all five tasks. Our results suggest that protein sequence models may benefit from leveraging biologically-inspired inductive biases that go beyond existing language modeling techniques in NLP.
This article introduces a novel protein structure alignment method (named TALI) based on the protein backbone torsion angle instead of the more traditional distance matrix. Because the structural alignment of the two proteins is based on the comparison of two sequences of numbers (backbone torsion angles), we can take advantage of a large number of well-developed methods such as Smith-Waterman or Needleman-Wunsch. Here we report the result of TALI in comparison to other structure alignment methods such as DALI, CE, and SSM ass well as sequence alignment based on PSI-BLAST. TALI demonstrated gr…
Toyota Technological Institute at Chicago（芝加哥丰田工业大学）
Proteins are the basic building blocks of life. They usually perform functions by folding to a particular structure. Understanding the folding process could help the researchers to understand the functions of proteins and could also help to develop supplemental proteins for people with deficiencies and gain more insight into diseases associated with troublesome folding proteins. Experimental methods are both expensive and time consuming. In this thesis I introduce a new machine learning based method to predict the protein structure. The new method improves the performance from two directions: …
Generative power of a protein language model trained on multiple sequence alignments
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Institute of Bioengineering（生物工程研究所）；School of Life Sciences（生命科学学院）；SIB Swiss Institute of Bioinformatics（瑞士生物信息学研究所）
Aligning Large Language Models and Geometric Deep Models for Protein Representation
Northwestern University（西北大学）；University of Pittsburgh（匹兹堡大学）；Michigan State University（密歇根州立大学）；North Carolina State University（北卡罗来纳州立大学）；New Jersey Institute of Technology（新泽西理工学院）
This work demonstrates how a complete statistical model quantifying the evolution of pairs of aligned proteins can be constructed from a time-parameterised substitution matrix and a time-parameterised 3-state alignment machine. All parameters of such a model can be inferred from any benchmark data-set of aligned protein sequences. This allows us to examine nine well-known substitution matrices on six benchmarks curated using various structural alignment methods; any matrix that does not explicitly model a "time"-dependent Markov process is converted to a corresponding base-matrix that does. In…
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
Tsinghua University（清华大学）；International Digital Economy Academy（国际数字经济学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Nanyang Technological University（南洋理工大学）
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；SIB Swiss Institute of Bioinformatics（瑞士生物信息学研究所）
Self-supervised neural language models with attention have recently been applied to biological sequence data, advancing structure, function and mutational effect prediction. Some protein language models, including MSA Transformer and AlphaFold's EvoFormer, take multiple sequence alignments (MSAs) of evolutionarily related proteins as inputs. Simple combinations of MSA Transformer's row attentions have led to state-of-the-art unsupervised structural contact prediction. We demonstrate that similarly simple, and universal, combinations of MSA Transformer's column attentions strongly correlate wit…
Enhancing the Protein Tertiary Structure Prediction by Multiple Sequence Alignment Generation
Shanghai AI Laboratory（上海人工智能实验室）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Zelixir Biotech（泽列西尔生物科技公司）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
The University of Hong Kong（香港大学）；Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；University of Waterloo（滑铁卢大学）
Protein language models have excelled in a variety of tasks, ranging from structure prediction to protein engineering. However, proteins are highly diverse in functions and structures, and current state-of-the-art models including the latest version of AlphaFold rely on Multiple Sequence Alignments (MSA) to feed in the evolutionary knowledge. Despite their success, heavy computational overheads, as well as the de novo and orphan proteins remain great challenges in protein representation learning. In this work, we show that MSAaugmented models inherently belong to retrievalaugmented methods. Mo…
Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction
Department of Systems Biology, Harvard Medical School（系统生物学系，哈佛医学院）；Department of Computer Science, University of Oxford（计算机科学系，牛津大学）；Department of Computer Science, University of Copenhagen（计算机科学系，哥本哈根大学）；Enzyme research, Novonesis（酶研究，Novonesis）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Sequence-based protein homology detection has been extensively studied and so far the most sensitive method is based upon comparison of protein sequence profiles, which are derived from multiple sequence alignment (MSA) of sequence homologs in a protein family. A sequence profile is usually represented as a position-specific scoring matrix (PSSM) or an HMM (Hidden Markov Model) and accordingly PSSM-PSSM or HMM-HMM comparison is used for homolog detection. This paper presents a new homology detection method MRFalign, consisting of three key components: 1) a Markov Random Fields (MRF) representa…
Boltzmann Machine Learning with a Parallel, Persistent Markov chain Monte Carlo method for Estimating Evolutionary Fields and Couplings from a Protein Multiple Sequence Alignment
Motivation Protein fold recognition is an important problem in structural bioinformatics. Almost all traditional fold recognition methods use sequence (homology) comparison to indirectly predict the fold of a tar get protein based on the fold of a template protein with known structure, which cannot explain the relationship between sequence and fold. Only a few methods had been developed to classify protein sequences into a small number of folds due to methodological limitations, which are not generally useful in practice. Results We develop a deep 1D-convolution neural network (DeepSF) to dire…
Evolutionary Profiles for Protein Fitness Prediction
Zhejiang University（浙江大学）；Peking University（北京大学）；East China University of Science and Technology（华东理工大学）；Chengdu University of Information Technology（成都信息工程大学）；Massachusetts Institute of Technology（麻省理工学院）；Zhejiang University of Technology（浙江工业大学）
Property-driven Protein Inverse Folding With Multi-Objective Preference Alignment
School of Computer Science, Peking University（北京大学计算机科学系）；BioGeometry；Mila - Québec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；HEC Montréal（蒙特利尔HEC商学院）；CIFAR AI Research Chair（CIFAR人工智能研究主席）
University of Oxford（牛津大学）；Harvard Medical School（哈佛医学院）；Cohere
The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease likelihood, to predicting immune-escape mutations in viruses and designing novel biotherapeutic proteins. Deep generative models of protein sequences trained on multiple sequence alignments have been the most successful approaches so far to address these tasks. The performance of these methods is however contingent on the availability of sufficiently deep and diverse alignments for reliable training. Their potential sc…
Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tencent AI Lab（腾讯人工智能实验室）；CryoEM Center, SUSTech（南方科技大学冷冻电镜中心）
Protein secondary structure prediction (PSSP) is essential for protein function analysis. However, for low homologous proteins, the PSSP suffers from insufficient input features. In this paper, we explicitly import external self-supervised knowledge for low homologous PSSP under the guidance of residue-wise profile fusion. In practice, we firstly demonstrate the superiority of profile over Position-Specific Scoring Matrix (PSSM) for low homologous PSSP. Based on this observation, we introduce the novel self-supervised BERT features as the pseudo profile, which implicitly involves the residue d…
Rapid progress in deep learning has spurred its application to bioinformatics problems including protein structure prediction and design. In classic machine learning problems like computer vision, progress has been driven by standardized data sets that facilitate fair assessment of new methods and lower the barrier to entry for non-domain experts. While data sets of protein sequence and structure exist, they lack certain components critical for machine learning, including high-quality multiple sequence alignments and insulated training / validation splits that account for deep but only weakly …
National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；Department of Computer Science and Technology（计算机科学与技术学院）；Nanjing University（南京大学）
ProFSA: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Information and Software Engineering, UESTC（电子科技大学信息与软件工程学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introd…
Mitochondrial diseases are currently untreatable due to our limited understanding of their pathology. We study the expression of various mitochondrial proteins in skeletal myofibres (SM) in order to discover processes involved in mitochondrial pathology using Imaging Mass Cytometry (IMC). IMC produces high dimensional multichannel pseudo-images representing spatial variation in the expression of a panel of proteins within a tissue, including subcellular variation. Statistical analysis of these images requires semi-automated annotation of thousands of SMs in IMC images of patient muscle biopsie…
