掩码预测的可识别性：模式盲性与掩码调度

中文摘要 AI 辅助

掩码预测通过拟合由调度加权的条件分布族来学习表示，但目前仍不清楚近最优条件预测何时能确定 underlying 的联合分布。我们针对具有两个分离良好的全局模式、且超出快速混合恢复保证范围的数据研究该问题，证明答案仅由掩码调度决定。在大上下文模式锚定下，对两个模式进行重加权可使联合分布在总变差上产生常数级变化，同时对可见上下文大小的掩码目标产生指数级微小扰动：由大上下文主导的掩码调度在理论上对全局模式权重盲性。为量化这一点，我们引入ε-可识别性模（即与给定超额风险一致的最大分布误差），并证明其在指数级小的超额风险下仍保持宏观尺度。精确信息分解明确了恢复可识别性的关键：模式权重敏感性由给定可见上下文的残余模式不确定性决定。因此，低可见度掩码可恢复该敏感性，且正的全掩码质量在不假设数据分布的所有可容许模型上锚定联合分布。在实验层面，我们从三个层级验证理论：对可计算分布的枚举验证了预测速率，梯度训练复现了模式盲性与恢复特性，对真实语料的测量将自然文本置于两种已验证的区间之间。

英文摘要

Masked prediction learns to infer missing variables from visible context selected by a mask schedule. When does small excess risk guarantee recovery of the true joint distribution? We study this question on finite product spaces under masked-block log loss, with conditionals induced by a single joint distribution. To quantify recovery, we introduce an $\varepsilon$-identifiability modulus measuring the worst-case error among joint distributions with population excess risk at most $\varepsilon$. For data with separated modes pinned down by sufficiently large visible contexts, schedules that always retain such contexts can permit nonvanishing mode-weight errors while incurring exponentially small excess risk. An information decomposition explains why: the loss captures mode-weight mismatch only where the visible context leaves the mode uncertain. We prove two-sided bounds showing that, over a fixed range of mode weights, sensitivity to mode reweighting is governed by residual mode uncertainty averaged over the mask schedule. Assigning schedule mass to low-visibility masks that retain this uncertainty yields recovery bounds within the reweighting family. Beyond this family, positive full-mask probability characterizes uniform control of joint KL divergence by excess risk. Exact calculations and controlled gradient optimization validate these predictions.

发表机构

• Australian Institute for Machine Learning（澳大利亚机器学习研究所）
• Adelaide University（阿德莱德大学）
• Responsible AI Research Centre（负责任人工智能研究中心）

机构由 AI 辅助整理，请以论文原文为准。