arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.17544cs.AIcs.CLcs.IR
收藏

通过可重用性和可验证性评估链式推理

Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

Indian Institute of Technology(印度理工学院)

Shashank Aggarwal, Ram Vikas Mishra, Amit Awekar

AI总结:

本文提出通过可重用性和可验证性评估链式推理的质量,发现专门推理模型并不始终优于通用大语言模型。

摘要:

在涉及搜索和排序等任务的多智能体信息检索管道中,基于大语言模型(LLM)的智能体会以链式推理(CoT)的形式相互交换中间推理过程。当前的CoT评估仅关注目标任务的准确性,然而这一指标无法评估推理过程本身的质量或实用性。为解决这一限制,我们引入了两个新的度量标准:可重用性和可验证性。我们通过Thinker-Executor框架将CoT生成与执行解耦。可重用性衡量执行者重用思考者CoT的难易程度,可验证性衡量执行者使用CoT匹配思考者答案的频率。我们评估了四个思考模型与十个执行模型在五个基准测试中的表现。我们的结果表明,可重用性和可验证性与标准准确性不相关,暴露出当前基于准确性的排行榜在推理能力评估上的盲点。令人惊讶的是,我们发现专门用于推理的模型生成的CoT并不总是比通用大语言模型(如Llama和Gemma)生成的更可重用或可验证。

英文摘要:

AI applications increasingly compose multiple models, where one model acts on the output produced by another model. The transferred text can also carry the reasoning of the sender model. Chain-of-thought is our experimental instance of such text. We ask three questions about such inter-model reasoning transfer. Is the reasoning transmitted reliably? Is it useful to the receiver? Is it safe? We study these three questions with a Sender-Receiver setup. A Sender LLM produces a reasoning trace. A separate Receiver LLM must act on it. Neither model knows the identity of the other. The Sender therefore cannot tailor its reasoning to a particular Receiver. Reliability asks whether the Receiver recovers the Sender's intended conclusion. We measure it with Verifiability. Usefulness asks whether transferred reasoning helps a Receiver correct its mistakes. We measure it with Correctability. Safety asks whether transferred reasoning makes a Receiver abandon a correct answer and follow the wrong reasoning supplied by the Sender. We measure it with Deceptiveness. All three factors (Task, Sender, and Receiver) affect all three metrics that we measure. However, there are some interesting trends. Nature of task is a more important factor in the reliability of reasoning transfer. Reliability is higher on arithmetic tasks and lower on science and commonsense tasks. The nature of Receiver plays more important role while deciding the usefulness of reasoning transfer. The nature of the Sender matters most for whether corrupted reasoning harms the Receiver.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑