纠正Dropout-LayerNorm期望差距以改进蛋白质结构模型

中文摘要 AI 辅助

尽管 $\mathbb{E}[\mathrm{dropout}(x)] = x$，但在这里我们表明 $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))]$ 并不等于 $\mathrm{LayerNorm}(x)$。因此，在许多基于AlphaFold2的蛋白质结构预测器中常见的Dropout层后接LayerNorm的模式，在评估时会产生系统性偏差，从而可能妨碍性能。为解决这一问题，我们推导出该差距的闭式一阶修正，称为Dropout-LayerNorm修正（DLC）。DLC在经验上与大型蒙特卡洛Dropout集成的性能提升相匹配。我们在九个蛋白质结构模型（ESMFold、OpenFold、ABB3、FlashABB、Ibex、NbForge、Genie1、Genie2、Genie3）上评估了其效果，涵盖成对抗体和单链抗体结构，以及一个蛋白质-配体对接模型（QuickBind）。该修正的计算开销可忽略不计，并在所有测试的十个模型中提高了准确性（约0.3%至13%），其中对ESMFold和OpenFold有适度但一致的改进，对抗体特异性模型有显著改进。这项工作揭示了将Dropout和LayerNorm串联起来的数学后果，并提供了一种免费、有原则的调整，以改善许多预训练模型的评估性能。复现实验的代码可在该https URL获取。

英文摘要

Although $\mathbb{E}[\mathrm{dropout}(x)] = x$, here we show that $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))]$ is not equal to $\mathrm{LayerNorm}(x)$. Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested ($\sim0.3\%-13\%$), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models. Code to reproduce the experiments is available at https://github.com/oxpig/DLC.

发表机构

• University of Oxford（牛津大学）
• Recursion（Recursion公司）

机构由 AI 辅助整理，请以论文原文为准。