通过反向学习推理步骤来衡量推理链的忠实性
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
AI总结：
通过反向学习推理步骤来评估推理链的参数忠实性，揭示模型推理与预测之间的关系。
摘要：
当提示语言模型（LMs）逐步思考时，模型会生成一个推理链（CoT），即模型 supposedly 用来生成其预测的推理步骤序列。尽管有许多关于CoT提示的工作，但不清楚在CoT中 verbalized 的推理是否忠实于模型的参数信念。我们引入了一个框架来衡量生成推理的参数忠实性，并提出了Faithfulness by Unlearning Reasoning steps（FUR），这是该框架的一个实例。FUR从模型参数中擦除推理步骤中的信息，并通过擦除对模型预测的影响来衡量忠实性。我们对四个LMs和五个多跳多选问答（MCQA）数据集的实验表明，FUR经常能够通过反向学习关键步骤精确地改变底层模型的预测，表明何时CoT参数上是忠实的。进一步分析显示，模型反向学习后生成的CoT支持不同的答案，暗示了反向学习的更深层次影响。
英文摘要：
When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
