DetectBench:大型语言模型能否检测并拼接隐式证据?
DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?
AI总结:
本文提出 DetectBench 基准,含 3,928 道题,验证 LLM 检测并拼接长上下文中隐式证据的能力,并设计侦探推理提示和微调方法,实验表明能有效提升模型证据检测及最终推理性能。
摘要:
在推理任务的过程中,检测上下文中的证据是一个关键步骤。评估和增强大型语言模型在证据检测方面的能力,将加强基于上下文的推理性能。本文提出了一个名为 DetectBench 的基准,用于验证在长上下文中检测并拼接隐式证据的能力。DetectBench 包含 3,928 道多项选择题,每道题平均包含 994 个标记。每道题平均包含 4.55 条隐式证据,解决该问题通常需要 7.62 次逻辑跳跃才能找到正确答案。为了增强大型语言模型在证据检测中的性能,本文提出了侦探推理提示和微调方法。实验表明,现有大型语言模型在长上下文中检测证据的能力远不如人类。然而,侦探推理提示有效增强了强大大型语言模型在证据检测方面的能力,而微调方法在增强较弱大型语言模型性能方面显示出显著效果。此外,当大型语言模型在证据检测方面的能力得到提升时,它们的最终推理性能也相应增强。
英文摘要:
Detecting evidence within the context is a key step in the process of reasoning task. Evaluating and enhancing the capabilities of LLMs in evidence detection will strengthen context-based reasoning performance. This paper proposes a benchmark called DetectBench for verifying the ability to detect and piece together implicit evidence within a long context. DetectBench contains 3,928 multiple-choice questions, with an average of 994 tokens per question. Each question contains an average of 4.55 pieces of implicit evidence, and solving the problem typically requires 7.62 logical jumps to find the correct answer. To enhance the performance of LLMs in evidence detection, this paper proposes Detective Reasoning Prompt and Finetune. Experiments demonstrate that the existing LLMs' abilities to detect evidence in long contexts are far inferior to humans. However, the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs. Moreover, when the abilities of LLMs in evidence detection are improved, their final reasoning performance is also enhanced accordingly.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。