引导式蛋白质语言模型中的流形外坍塌
Off-Manifold Collapse in Guided Protein Language Models
浏览论文内容
中文总结 AI 辅助
该研究发现引导式蛋白质语言模型存在流形外坍塌问题，提出无需训练的马氏过滤方法，可低成本提升生成序列的属性得分与结构合理性，且可跨引导方法迁移。
中文摘要 AI 辅助
蛋白质语言模型是蛋白质序列设计中广泛使用的先验模型，越来越多的研究在推理阶段对其进行控制，作为微调的替代方案。这种引导面临一个两难：引导强度足够温和以保留自然激活统计量时，几乎无法改变所需属性；引导强度足够强以改变属性时，生成的序列越来越难以折叠。我们表明这种失败具有特定且可低成本检测的特征，即模型自身表示的流形外坍塌。引导后的激活会向与随机氨基酸输入统计上无法区分的区域移动，序列退化为低复杂度，然而被优化的属性预言机仍可将这些生成结果判定为成功。因此，优化后的预言机可能无法察觉这种坍塌，对于溶解性任务甚至会主动奖励这种坍塌，而结构和组成属性则会暴露该失败。由于该失败在候选序列生成完成后即可显现，我们在输出端检测而非修改生成器。我们引入一种针对天然蛋白质激活的低成本密度先验，仅保留在该先验下保持典型性的候选序列，这一无需训练的事后步骤称为马氏过滤（Mahalanobis filtering）。在匹配的引导设置下，该方法以可忽略的成本同时提升了保留候选序列的属性得分和结构合理性，且不触及生成器，可跨不同引导方法迁移。我们在该 https URL 发布了激活统计量。
英文摘要
Protein language models are widely used priors for protein sequence design, and a growing body of work controls them at inference time as an alternative to fine-tuning. Such guidance faces a dilemma: mild enough to preserve natural activation statistics, it barely moves the property; strong enough to move it, the generations become progressively harder to fold. We show the failure has a specific and cheaply detectable signature, an off-manifold collapse of the model's own representations. Guided activations fall toward a region statistically indistinguishable from random amino-acid input, and the sequences degenerate to low complexity, yet the property oracle being optimized can still score these generations as a success. The optimized oracle can therefore fail to witness the collapse and, for solubility, can actively reward it, whereas structure and composition expose the failure. Because the failure is already visible in a finished candidate, we detect it at the output rather than modify the generator. We introduce a cheap density prior over natural protein activations and keep only the candidates that remain typical under it, a training-free post-hoc step we call Mahalanobis filtering. At matched guidance settings it improves both the property score and the structural plausibility of the sequences it keeps at negligible cost, without touching the generator, and transfers across different guidance methods. We release the activation statistic at https://huggingface.co/Shuibai12138/off-manifold-collapse-plm
发表机构
- Duke University（杜克大学）
- Cornell University（康奈尔大学）
- University of Wisconsin--Madison（威斯康星大学麦迪逊分校）
机构由 AI 辅助整理，请以论文原文为准。
