We present protein autoregressive modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation via coarse-to-fine next-scale prediction. Using the hierarchical nature of proteins, PAR generates structures that mimic sculpting a statue, forming a coarse topology and refining structural details over scales. To achieve this, PAR consists of three key components: (i) multi-scale downsampling operations that represent protein structures across multiple scales during training; (ii) an autoregressive transformer that encodes multi-scale information and produces conditional embeddings to guide structure generation; (iii) a flow-based backbone decoder that generates backbone atoms conditioned on these embeddings. Moreover, autoregressive models suffer from exposure bias, caused by the training and the generation procedure mismatch, and substantially degrades structure generation quality. We effectively alleviate this issue by adopting noisy context learning and scheduled sampling, enabling robust backbone generation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible human-prompted conditional generation and motif scaffolding without requiring fine-tuning. On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality, and exhibits favorable scaling behavior. Together, these properties establish PAR as a promising framework for protein structure generation.
University of Southern California（南加州大学）；Hood College（胡德学院）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Machine learning (ML) is revolutionizing protein structural analysis, including an important subproblem of predicting protein residue contact maps, i.e., which amino-acid residues are in close spatial proximity given the amino-acid sequence of a protein. Despite recent progresses in ML-based protein contact prediction, predicting contacts with a wide range of distances (commonly classified into short-, medium- and long-range contacts) remains a challenge. Here, we propose a multiscale graph neural network (GNN) based approach taking a cue from multiscale physics simulations, in which a standar…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Capturing intricate biological phenomena often requires multiscale modeling where coarse and inexpensive models are developed using limited components of expensive and high-fidelity models. Here, we consider such a multiscale framework in the context of cancer biology and address the challenge of evaluating the descriptive capabilities of a continuum model developed using 1-dimensional statistics from a molecular dynamics model. Using deep learning, we develop a highly predictive classification model that identifies complex and emergent behavior from the continuum model. With over 99.9% accura…
Towards Multiscale Graph-based Protein Learning with Geometric Secondary Structural Motifs
Department of Mathematics and Scientific Computing and Imaging (SCI) Institute University of Utah（数学与科学计算与成像研究所（SCI）大学）；Department of Mathematics, UCLA（数学系，加州大学洛杉矶分校）；Department of Mathematics, UC Davis（数学系，加州大学戴维斯分校）
DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Intel（英特尔公司）；HEC Montréal（蒙特利尔高等商学院）；CIFAR AI Research Chair（加拿大高级研究所人工智能研究主席）
LightOn（LightOn公司）；University of Oxford（牛津大学）；Harvard Medical School（哈佛医学院）
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Such generative models hold the promise of greatly accelerating protein design. We conduct the first systematic study of how capabilities evolve with model size for autoregressive transformers in the protein domain: we evaluate RITA models in next amino acid prediction, zero-shot fitness, and enzyme function prediction, showing benefits from increased scale. We release the RITA m…
University of Oxford（牛津大学）；Harvard Medical School（哈佛医学院）；Cohere
The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease likelihood, to predicting immune-escape mutations in viruses and designing novel biotherapeutic proteins. Deep generative models of protein sequences trained on multiple sequence alignments have been the most successful approaches so far to address these tasks. The performance of these methods is however contingent on the availability of sufficiently deep and diverse alignments for reliable training. Their potential sc…
Hotspot-Driven Peptide Design via Multi-Fragment Autoregressive Extension
Tsinghua University（清华大学）；University of Washington（华盛顿大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；ByteDance Inc.（字节跳动公司）；Helixon Inc.（Helixon公司）
Uni-Flow: a unified autoregressive-diffusion model for complex multiscale flows
Centre for Computational Science, University College London, London, UK；Department of Earth Science；Engineering, Imperial College London, London, UK；Department of Chemical Engineering, University College London, London, UK；Department of Physics, Eindhoven University of Technology, Eindhoven, Netherlands；School of Civil \& Environmental Engineering, Queensland University of Technology, Brisbane, Australia；Australian Centre for Water；Environmental Biotechnology, The University of Queensland, Brisbane, Australia；Institute for Mechanics, Computational Mechanics Group, Technical University of Darmstadt, Germany；Centre for Advanced Research Computing, University College London, London, UK
Protein secondary structure prediction is an important problem in bioinformatics. Inspired by the recent successes of deep neural networks, in this paper, we propose an end-to-end deep network that predicts protein secondary structures from integrated local and global contextual features. Our deep architecture leverages convolutional neural networks with different kernel sizes to extract multiscale local contextual features. In addition, considering long-range dependencies existing in amino acid sequences, we set up a bidirectional neural network consisting of gated recurrent unit to capture g…
University of Washington（华盛顿大学）；Salesforce Research（赛富时研究院）
For protein sequence datasets, unlabeled data has greatly outpaced labeled data due to the high cost of wet-lab characterization. Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. However, the optimal pre-training strategy remains an open question. Instead of strictly borrowing from natural language processing (NLP) in the form of masked or autoregressive language modeling, we introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments.…
BioGeometry；Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；HEC Montréal（蒙特利尔高等商学院）
Proteins are macromolecules that perform essential functions in all living organisms. Designing novel proteins with specific structures and desired functions has been a long-standing challenge in the field of bioengineering. Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models, both of which suffer from high inference costs. In this paper, we propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state from random initializati…
Motivation: Understanding functions of proteins in specific human tissues is essential for insights into disease diagnostics and therapeutics, yet prediction of tissue-specific cellular function remains a critical challenge for biomedicine. Results: Here we present OhmNet, a hierarchy-aware unsupervised node feature learning approach for multi-layer networks. We build a multi-layer network, where each layer represents molecular interactions in a different human tissue. OhmNet then automatically learns a mapping of proteins, represented as nodes, to a neural embedding based low-dimensional spac…
Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling
DP Technology（DP技术公司）；AI for Science Institute（人工智能科学研究院）；School of Mathematical Sciences（数学科学学院）；Center for Machine Learning Research（机器学习研究中心）；Institute for Artificial Intelligence（人工智能研究院）
Learning from Hetero Density for Cryo-EM Protein Reconstruction
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Ant Group（蚂蚁集团）
Université Paris-Saclay（巴黎-萨克雷大学）；CNRS（法国国家科学研究中心）；CentraleSupelec（中央理工-高等电力学院）；Laboratoire des Signaux et Systèmes（信号与系统实验室）
We propose a practical deep generative approach for lossless point cloud geometry compression, called MSVoxelDNN, and show that it significantly reduces the rate compared to the MPEG G-PCC codec. Our previous work based on autoregressive models (VoxelDNN) has a fast training phase, however, inference is slow as the occupancy probabilities are predicted sequentially, voxel by voxel. In this work, we employ a multiscale architecture which models voxel occupancy in coarse-to-fine order. At each scale, MSVoxelDNN divides voxels into eight conditionally independent groups, thus requiring a single n…
Poisoning the corpora of protein design systems during pre-training and retrieval
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；BSRC “Alexander Fleming,”（亚历山大·弗莱明基础科学研究中心）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
Partner-aware Peptide-Protein Interaction Prediction and Target-conditioned Peptide Generation
School of Mathematics and Statistics, Lanzhou University（兰州大学数学与统计学院）；Cuiying Honors College, Lanzhou University（兰大优培学院）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
