从词到氨基酸：深度诅咒是否依然存在？
From Words to Amino Acids: Does the Curse of Depth Persist?
AI总结：
本文研究蛋白质语言模型中深度效率问题，通过分析七种主流PLM家族，发现任务相关计算集中在部分层，其余层主要提供预测细化，这一趋势在序列和多模态模型中均存在。
摘要：
蛋白质语言模型（PLMs）已广泛应用于蛋白质工程和从头设计，通常通过训练深度Transformer模型进行下一步或掩码token预测。最近研究表明，自回归LLM中存在深度诅咒：许多后期层对最终输出预测贡献很小。本文分析了七种流行PLM家族在不同模型规模下的深度效率，通过统一的探测、扰动和下游评估测量，量化了层贡献随深度的变化。在不同模型中观察到一致的深度依赖模式，这些模式扩展了LLM的研究成果：大量任务相关计算集中在部分层，其余层主要提供最终预测的逐步细化。这些趋势不仅存在于序列-only设置中，也出现在多模态PLMs中。总体而言，我们的结果表明深度效率低下是现代PLMs的普遍特征，推动了未来更高效的架构和训练方法的研究。
英文摘要：
Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
