Various approaches utilizing Transformer architectures have achieved state-of-the-art results in Natural Language Processing (NLP). Based on this success, numerous architectures have been proposed for other types of data, such as in biology, particularly for protein sequences. Notably among these are the ESM2 architectures, pre-trained on billions of proteins, which form the basis of various state-of-the-art approaches in the field. However, the ESM2 architectures have a limitation regarding input size, restricting it to 1,022 amino acids, which necessitates the use of preprocessing techniques to handle sequences longer than this limit. In this paper, we present the long and quantized versions of the ESM2 architectures, doubling the input size limit to 2,048 amino acids.
Linear-scaling kernels for protein sequences and small molecules outperform deep learning while providing uncertainty quantitation and improved interpretability
Reverse Distillation: Consistently Scaling Protein Language Model Representations
Department of Computer Science(计算机科学系);Duke University(杜克大学);Center for Computational Biology(计算生物学中心);Flatiron Institute(Flatiron研究所);Depts of Biostats. and Bioinformatics(生物统计学与生物信息学系;细胞生物学系);and Cell Biology
Data-Error Scaling Laws in Machine Learning on Combinatorial Mutation-prone Sets: Proteins and Small Molecules
University of Basel(巴塞尔大学);ETH Zurich(苏黎世联邦理工学院);Swiss Nanoscience Institute(瑞士纳米科学研究所);University of Toronto(多伦多大学);Vector Institute(向量研究所);TU Berlin(柏林技术大学)
Scaling Atomistic Protein Binder Design with Generative Pretraining and Test-Time Compute
NVIDIA(英伟达);University of Oxford(牛津大学);Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);HEC Montréal(蒙特利尔高等商学院);CIFAR AI Chair(CIFAR人工智能主席);AITHYRA;School of Biological Sciences, Seoul National University(首尔大学生物科学学院);Interdisciplinary Program in Bioinformatics, Seoul National University(首尔大学生物信息学跨学科项目);Institute of Molecular Biology and Genetics, Seoul National University(首尔大学分子生物学与遗传学研究所);Artificial Intelligence Institute, Seoul National University(首尔大学人工智能研究所)
Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所认知与决策智能复杂系统重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);College of Computer Science, Inner Mongolia University(内蒙古大学计算机学院)