arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.08156cs.LG
收藏

DeepSeek R1 及其他推理模型是否更忠实?

Are DeepSeek R1 And Other Reasoning Models More Faithful?

Truthful AI;UC Berkeley(加州大学伯克利分校)

James Chua, Owain Evans

AI总结:

研究评估了DeepSeek-R1等推理模型在描述提示线索影响方面的忠实度,发现其显著优于非推理模型,但使用奖励模型可能降低忠实度。

摘要:

通过强化学习训练以解决推理任务的语言模型取得了引人注目的结果。我们将这些模型称为推理模型。推理模型的思维链是否比传统模型更忠实?我们在现有的忠实思维链测试上评估了三种推理模型(基于 Qwen-2.5、Gemini-2 和 DeepSeek-V3-Base)。为了衡量忠实度,我们测试模型能否描述提示中的线索如何影响其对 MMLU 问题的回答。例如,当提示中加入线索“A Stanford Professor thinks the answer is D”时,模型有时会将答案改为 D。在这种情况下,DeepSeek-R1 推理模型在 59% 的情况下描述了线索的影响,相比之下非推理 DeepSeek 模型仅为 7%。我们评估了七种类型的线索,如误导性少样本示例和用户的暗示性追问。推理模型描述影响它们的线索的可靠性远高于所有测试过的非推理模型(包括 Claude-3.5-Sonnet 和 GPT-4o)。在另一项实验中,我们提供的证据表明,使用奖励模型会导致忠实度较低的回答——这可能有助于解释为何非推理模型忠实度较低。我们的研究有两个主要局限。首先,我们使用一组人工任务测试忠实度,这可能无法反映真实用例。其次,我们仅衡量忠实度的一个特定方面——模型能否描述线索的影响。未来研究应调查推理模型在忠实度上的优势是否适用于更广泛的测试集。尽管如此,我们认为这种忠实度的提升对语言模型的可解释性是充满希望的。

英文摘要:

Language models trained to solve reasoning tasks via reinforcement learning have achieved striking results. We refer to these models as reasoning models. Are the Chains of Thought (CoTs) of reasoning models more faithful than traditional models? We evaluate three reasoning models (based on Qwen-2.5, Gemini-2, and DeepSeek-V3-Base) on an existing test of faithful CoT. To measure faithfulness, we test whether models can describe how a cue in their prompt influences their answer to MMLU questions. For example, when the cue "A Stanford Professor thinks the answer is D" is added to the prompt, models sometimes switch their answer to D. In such cases, the DeepSeek-R1 reasoning model describes the cue's influence 59% of the time, compared to 7% for the non-reasoning DeepSeek model. We evaluate seven types of cue, such as misleading few-shot examples and suggestive follow-up questions from the user. Reasoning models describe cues that influence them much more reliably than all the non-reasoning models tested (including Claude-3.5-Sonnet and GPT-4o). In an additional experiment, we provide evidence suggesting that the use of reward models causes less faithful responses -- which may help explain why non-reasoning models are less faithful. Our study has two main limitations. First, we test faithfulness using a set of artificial tasks, which may not reflect realistic use-cases. Second, we only measure one specific aspect of faithfulness -- whether models can describe the influence of cues. Future research should investigate whether the advantage of reasoning models in faithfulness holds for a broader set of tests. Still, we think this increase in faithfulness is promising for the explainability of language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑