CiPO：通过迭代偏好优化实现大型推理模型的反事实去学习
CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization
AI总结：
针对大型推理模型去学习难题，CiPO通过迭代偏好优化重新定义去学习为对推理过程的针对性干预，有效去除中间推理步骤和最终答案的知识，同时保持推理能力。
摘要：
近年来，机器去学习技术因其能有效从大规模人类数据训练的大型语言模型中选择性删除 unwanted privacy 或版权信息而受到越来越多关注。然而，大型推理模型（LRMs）的出现，其强调长链推理（CoT）来解决复杂问题，给去学习带来了挑战：现有方法要么难以完全消除不利知识，要么因干扰推理过程而降低推理性能。为此，我们引入了通过迭代偏好优化的反事实去学习（CiPO）框架，重新定义去学习为对LRMs的CoT推理过程的针对性干预。具体而言，给定一个期望的去学习目标答案，CiPO指导LRMs生成一个逻辑有效的反事实推理轨迹以进行偏好调优。随着LRM适应反事实轨迹，CiPO迭代更新偏好学习数据以增加与原始模型的差异。这种迭代循环确保了理想的去学习和流畅的优化，有效缓解了这一难题。在具有挑战性的基准测试中，实验表明CiPO在去学习方面表现出色，能够完全去除中间Co T步骤和最终答案的知识，同时保持LRMs的推理能力。
英文摘要：
Machine unlearning has gained increasing attention in recent years, as a promising technique to selectively remove unwanted privacy or copyrighted information from Large Language Models that are trained on a massive scale of human data. However, the emergence of Large Reasoning Models (LRMs), which emphasize long chain-of-thought (CoT) reasoning to address complex questions, presents a dilemma to unlearning: existing methods either struggle to completely eliminate undesired knowledge from the CoT traces or degrade the reasoning performances due to the interference with the reasoning process. To this end, we introduce Counterfactual Unlearning through iterative Preference Optimization (CiPO), a novel framework that redefines unlearning as the targeted intervention of the CoT reasoning in LRMs. More specifically, given a desired unlearning target answer, CiPO instructs LRMs to generate a logically valid counterfactual reasoning trace for preference tuning. As the LRM adjusts to the counterfactual trace, CiPO iteratively updates the preference learning data to increase the discrepancy from the original model. This iterative loop ensures both desirable unlearning and smooth optimization, effectively mitigating the dilemma. Experiments on challenging benchmarks demonstrate that CiPO excels at unlearning, completely removing knowledge from both the intermediate CoT steps and the final answer, while preserving the reasoning abilities of LRMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
