arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06475cs.LG
收藏

迈向通用性推理:面向LLM推理的群体因果反事实策略优化

Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning

Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Wangxuan Institute of Computer Technology, Peking University(北京大学王宣计算机技术研究所);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Jingyao Wang, Peizheng Guo, Wenwen Qiang, Jiahuan Zhou, Huijie Guo, Changwen Zheng, Hui Xiong

AI总结:

本文提出群体因果反事实策略优化方法,通过反事实奖励提升LLM推理的通用性,强调推理过程的鲁棒性和有效性,实验验证其有效性。

摘要:

大型语言模型(LLMs)在复杂任务中表现出色,但现有奖励机制过于关注最终正确性,忽视推理过程。本文从因果视角出发,将多候选推理视为反事实实验,提出群体因果反事实策略优化方法,通过反事实奖励训练LLM学习通用推理模式,提升推理的鲁棒性和有效性。实验表明该方法在多样化基准上具有优势。

英文摘要:

Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities. However, existing reward mechanisms remain tightly coupled to final correctness and pay little attention to the underlying reasoning process: trajectories with sound reasoning but wrong answers receive low credit, while lucky guesses with flawed logic may be highly rewarded, affecting reasoning generalization. From a causal perspective, we interpret multi-candidate reasoning for a fixed question as a family of counterfactual experiments with theoretical supports. Building on this, we propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns. It proposes an episodic causal counterfactual reward that jointly captures (i) robustness, encouraging the answer distribution induced by a reasoning step to remain stable under counterfactual perturbations; and (ii) effectiveness, enforcing sufficient variability so that the learned reasoning strategy can transfer across questions. We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust. Extensive experiments on diverse benchmarks demonstrate its advantages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑