因果增强的强化策略优化
Causally-Enhanced Reinforcement Policy Optimization
AI总结：
提出 CE-PO 奖励塑形框架，利用 Jacobian 敏感度与反事实强化衡量 Z→X→Y 因果连贯性，并兼容 PPO/GRPO，在4个数据集上平均提升准确率5.49%且增强因果鲁棒性。
摘要：
使用强化目标训练的大型语言模型（LLM）通常会通过捷径策略获得表面上正确的答案，即将正确输出与虚假或不忠实的推理配对，并在微小因果扰动下性能下降。我们提出了因果增强策略优化（Causally-Enhanced Policy Optimization，CE-PO），这是一种即插即用的奖励塑形框架，通过沿从提示（Z）到推理依据（X）再到答案（Y）的生成路径加入因果连贯性的可微代理，从而增强策略优化。CE-PO 使用基于 Jacobian 的敏感度估计模型内部影响，对这些信号进行反事实强化以抑制干扰线索，并通过 Minkowski（幂均值）组合器将所得连贯性分数与任务准确性反馈融合，暴露出一个可调参数，用于平衡准确性与连贯性。该统一奖励无需架构更改即可与 PPO/GRPO 集成。在推理基准和因果压力测试中，CE-PO 减少了奖励黑客行为和不忠实的思维链，同时提高了对相关—因果翻转及轻微反事实编辑的鲁棒性，且准确性几乎持平。跨4个数据集的实验结果表明，CE-PO 相比基线平均提高准确性5.49%（最高达9.58%），同时提升了对相关—因果翻转和轻微反事实编辑的鲁棒性。
英文摘要：
Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small causal perturbations. We introduce Causally-Enhanced Policy Optimization (CE-PO), a drop-in reward-shaping framework that augments policy optimization with a differentiable proxy for causal coherence along the generation pathway from prompt (Z) to rationale (X) to answer (Y). CE-PO estimates model-internal influence with Jacobian-based sensitivities, counterfactually hardens these signals to suppress nuisance cues, and fuses the resulting coherence score with task-accuracy feedback via a Minkowski (power-mean) combiner, exposing a single tunable between accuracy and coherence trade-off. The unified reward integrates with PPO/GRPO without architectural changes. Across reasoning benchmarks and causal stress tests, CE-PO reduces reward hacking and unfaithful chain-of-thought while improving robustness to correlation-causation flips and light counterfactual edits, all at near-parity accuracy. Experimental results across 4 datasets show that CE-PO improves accuracy over baselines by 5.49% on average (up to 9.58%), while improving robustness to correlation-causation flips and light counterfactual edits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
