Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
Google（谷歌）；University of Cambridge（剑桥大学）；Alan Turing Institute（艾伦·图灵研究所）；DeepMind（深度思维）
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, Adrian Weller
备注：
This arXiv submission has been deprecated. Please see "Rethinking Attention with Performers" at arXiv:2009.14794 for the most updated version of the paper
Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical analysis.
Joint Masked Reconstruction and Contrastive Learning for Mining Interactions Between Proteins
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；Institute of Artificial Intelligence, Huazhong University of Science and Technology（华中科技大学人工智能研究院）；Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology（华中科技大学湖北省类脑智能系统重点实验室）；Key Laboratory of Image Processing and Intelligent Control (Huazhong University of Science and Technology), Ministry of Education（华中科技大学图像信息处理与智能控制教育部重点实验室）
TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models
Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）
3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health（亥姆霍兹慕尼黑中心 - 德国环境健康研究中心，计算健康中心，健康人工智能研究所与亥姆霍兹人工智能）；Department of Medicine III, Ludwig-Maximilian-University Hospital（路德维希-马克西米利安大学医院第三内科）；Department of Physics, Ludwig-Maximilian-University（路德维希-马克西米利安大学物理系）；German Cancer Consortium (DKTK), partner site Munich（德国癌症联盟（DKTK）慕尼黑合作站点）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction
Université de Montréal（蒙特利尔大学）；Mila（米拉计算科学研究所）；Dreamfold（Dreamfold公司）；Duke University（杜克大学）；McGill University（麦吉尔大学）；Allen Institute for AI（艾伦人工智能研究所）；Oxford University（牛津大学）；Aithyra（Aithyra公司）
Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training
Harvard Medical School（哈佛医学院）；University of Copenhagen（哥本哈根大学）；Novo Nordisk A/S（诺和诺德公司）；Microsoft Research（微软研究院）；Technical University of Denmark（丹麦技术大学）
Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information
Seoul National University（首尔国立大学）；LG AI Research（LG人工智能研究院）；NAVER Corp.（NAVER公司）；Kangwon National University（江原国立大学）；Seoul National University of Science and Technology（首尔科技大学）
ProteinGuide: On-the-fly property guidance for protein sequence generative models
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley（电气工程与计算机科学系，加州大学伯克利分校）；Center for Computational Biology, University of California, Berkeley（计算生物学中心，加州大学伯克利分校）；Department of Molecular and Cell Biology, University of California, Berkeley（分子与细胞生物学系，加州大学伯克利分校）；Innovative Genomics Institute, University of California, Berkeley（创新基因组研究所，加州大学伯克利分校）；Howard Hughes Medical Institute, University of California, Berkeley（霍华德·休斯医学研究所，加州大学伯克利分校）
Generative power of a protein language model trained on multiple sequence alignments
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Institute of Bioengineering（生物工程研究所）；School of Life Sciences（生命科学学院）；SIB Swiss Institute of Bioinformatics（瑞士生物信息学研究所）
Evolutionary Profiles for Protein Fitness Prediction
Zhejiang University（浙江大学）；Peking University（北京大学）；East China University of Science and Technology（华东理工大学）；Chengdu University of Information Technology（成都信息工程大学）；Massachusetts Institute of Technology（麻省理工学院）；Zhejiang University of Technology（浙江工业大学）
From Words to Amino Acids: Does the Curse of Depth Persist?
Technical University of Munich（慕尼黑技术大学）；Helmholtz AI, Munich（慕尼黑海德堡研究所人工智能部门）；KTH Royal Institute of Technology（皇家理工学院）；Institute of Computational Biology, Helmholtz Munich（海德堡慕尼黑计算生物学研究所）；Google, Paradigms of Intelligence Team（谷歌，智能范式团队）
Structure Language Models for Protein Conformation Generation
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；University of Ottawa（渥太华大学）；National Research Council Canada（加拿大国家研究委员会）；CIFAR AI Chair（加拿大高级研究所人工智能主席项目）；HEC Montréal（蒙特利尔高等商学院）
Linear Fitness Subspace in Protein Language Models Enables Sample-Efficient Directed Evolution
Nanyang Technological University（南洋理工大学）；Sun Yat-sen University（中山大学）；Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；Shanghai Institute of Optics and Fine Mechanics（上海光学精密机械研究所）；Zhongnan Hospital, Wuhan University（武汉大学中南医院）；University of Southern California（南加州大学）
Linear-scaling kernels for protein sequences and small molecules outperform deep learning while providing uncertainty quantitation and improved interpretability
In-Context Learning can distort the relationship between sequence likelihoods and biological fitness
Computational Biology and Bioinformatics Program（计算生物学与生物信息学项目）；Emeritus Department of Ecology and Evolutionary Biology（退休生态学与进化生物学系）；Department of Evolutionary Biology（进化生物学系）；Hagler Institute for Advanced Studies（哈格勒高级研究 institute）；Department of Physics（物理学系）；Quantitative Biology Institute（定量生物学研究所）
Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling
DP Technology（DP技术公司）；AI for Science Institute（人工智能科学研究院）；School of Mathematical Sciences（数学科学学院）；Center for Machine Learning Research（机器学习研究中心）；Institute for Artificial Intelligence（人工智能研究院）
