arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.05787cs.CL
收藏

证据链与证据生成:面向基于上下文扎根和检索增强推理的提示方法

Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoning

Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所)

Md Rizwan Parvez

AI总结:

本文提出单/双步零样本 CoE 与 E2G 提示框架,通过抽取上下文中的显式证据约束 LLM 推理,在 LogiQA 和 DROP 上取得优于 CoT、ToT、CR 及 Gemini Ultra 的结果。

摘要:

尽管思维链(CoT)提示已经彻底改变了 LLM 执行推理任务的方式,但其当前方法及变体(例如 Self-consistency、ReACT、Reflexion、Tree-of-Thoughts(ToT)、Cumulative Reasoning(CR)等)仍受到上下文扎根不足、幻觉或输出生成不一致,以及迭代迟缓等限制。为克服这些挑战,我们提出一种新颖的单步/双步零样本提示框架,该框架建立在两个独特策略之上:Chain of Evidences(CoE,证据链)和 Evidence to Generate(E2G,证据生成)。与未经证实的推理主张不同,我们的创新方法利用“证据用于决策”的力量:首先仅关注上下文中明确提到的思维序列,这些序列随后作为抽取的证据,以更高的精确性和效率指导 LLM 的输出生成过程。这一简单而有效的方法释放了思维链提示的全部潜力,促进 LLM 实现更快、更可靠且具备上下文感知能力的推理。我们的框架在各类知识密集型推理和生成任务上持续取得显著结果,并在使用最先进 LLM 时超越基线方法。例如:(i)在使用 GPT-4 的 LogiQA 基准上,CoE 达到 53.8% 的最新最先进准确率,分别比 CoT 高 18%、比 ToT 高 11%、比 CR 高 9%;(ii)结合 PaLM-2 的 CoE 在 DROP 上取得 83.3 的 F1 分数,比 Gemini Ultra 的可变样本性能高出 0.9 个 F1 点。我们将在这些基准上的提示和输出作为新的指令微调数据集发布,以供未来研究使用,地址为 https://huggingface.co/datasets/kagnlp/Chain-of-Evidences/。

英文摘要:

While chain-of-thoughts (CoT) prompting has revolutionized how LLMs perform reasoning tasks, its current methods and variations (e.g, Self-consistency, ReACT, Reflexion, Tree-of-Thoughts (ToT), Cumulative Reasoning (CR) etc.,) suffer from limitations like limited context grounding, hallucination/inconsistent output generation, and iterative sluggishness. To overcome these challenges, we introduce a novel mono/dual-step zero-shot prompting framework built upon two unique strategies Chain of Evidences (CoE)} and Evidence to Generate (E2G). Instead of unverified reasoning claims, our innovative approaches leverage the power of "evidence for decision making" by first focusing exclusively on the thought sequences explicitly mentioned in the context which then serve as extracted evidence, guiding the LLM's output generation process with greater precision and efficiency. This simple yet potent approach unlocks the full potential of chain-of-thoughts prompting, facilitating faster, more reliable, and contextually aware reasoning in LLMs. Our framework consistently achieves remarkable results across various knowledge-intensive reasoning and generation tasks, surpassing baseline approaches with state-of-the-art LLMs. For instance, (i) on the LogiQA benchmark using GPT-4, CoE achieves a new state-of-the-art accuracy of 53.8%, surpassing CoT by 18%, ToT by 11%, and CR by 9%; (ii) CoE with PaLM-2 outperforms the variable-shot performance of Gemini Ultra by 0.9 F1 points, achieving an F1 score of 83.3 on DROP. We release our prompts and outputs on these benchmarks as a new instruction tuning dataset for future research at https://huggingface.co/datasets/kagnlp/Chain-of-Evidences/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑