发表机构
School of Artificial Intelligence, University of Chinese Academy of Sciences; Institute of Information Engineering, Chinese Academy of Sciences（中国科学院大学人工智能学院; 中国科学院信息工程研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对不完美信息学习中耦合展开的噪声与安全问题，提出观测安全反事实耦合框架及OSCC-Select选择器，通过认证式耦合优化显著降低策略梯度噪声方差。
AI 中文摘要
耦合展开可以减少反事实动作比较中的噪声，但两个问题阻碍了标准公共随机数构造在不完美信息环境中作为通用学习原语。首先，无效耦合可能暴露隐藏状态、同步内生策略随机性或在反事实历史分叉后错配机会事件。其次，在多动作策略优化中，较低的回报对比方差本身并非相关目标：优化器依赖于通过局部策略梯度几何投影后的回报协方差矩阵。我们引入了观测安全反事实耦合（OSCC），该框架通过边际保持、信息状态安全、分支局部策略随机性、语义事件对齐和迹先于预言机重放来定义可接受类别。我们推导出一个梯度感知的耦合准则，表明对于边际保持耦合，策略梯度噪声变化由策略雅可比加权的非对角回报协方差决定。这促使了OSCC-Select，一个仅需校准的选择器，它使用独立的安全和增益证书在独立、仅根、仅延续和完全耦合展开之间进行选择。其增益目标结合了投影梯度噪声与测量的物理采样成本，并且每当同时下置信界未证明改进时，便回退到独立采样。在100,000次固定根Leduc比较中，完全耦合的CP-GRPO实例将回报对比方差从41.1158降至18.1441，降低了55.87%，同时保持了声明的分支边际。对于三个动作，OSCC-Select选择延续耦合，并达到梯度噪声迹0.0783，而回报方差选择为0.0917。将校准从64组增加到2,048组，认证从0.327提高到0.995。
英文摘要
Coupled rollouts can reduce the noise of counterfactual action comparisons, but two issues prevent standard common-random-number constructions from serving as a general learning primitive in imperfect-information environments. First, an invalid coupling may expose hidden state, synchronize endogenous policy randomness, or misalign chance events after counterfactual histories diverge. Second, in multi-action policy optimization, lower return-contrast variance is not by itself the relevant objective: the optimizer depends on the return covariance matrix after projection through the local policy-gradient geometry. We introduce observation-safe counterfactual coupling (OSCC), a framework that defines an admissible class through marginal preservation, information-state safety, branch-local policy randomness, semantic event alignment, and trace-before-oracle replay. We derive a gradient-aware coupling criterion showing that, for marginal-preserving couplings, policy-gradient noise changes are determined by policy-Jacobian-weighted off-diagonal return covariance. This motivates OSCC-Select, a calibration-only selector that chooses among independent, root-only, continuation-only, and fully coupled rollouts using separate safety and gain certificates. Its gain target combines projected gradient noise with measured physical sampling cost and falls back to independent sampling whenever a simultaneous lower confidence bound does not certify improvement. On 100,000 fixed-root Leduc comparisons, the fully coupled CP-GRPO instantiation reduces return-contrast variance from 41.1158 to 18.1441, a 55.87% reduction, while preserving the declared branch marginals. With three actions, OSCC-Select chooses continuation coupling and attains gradient-noise trace 0.0783 versus 0.0917 for return-variance selection. Increasing calibration from 64 to 2,048 groups raises certification from 0.327 to 0.995.
Comments34 pages, 10 figures
