尽管思维链(CoT)提示已经彻底改变了 LLM 执行推理任务的方式,但其当前方法及变体(例如 Self-consistency、ReACT、Reflexion、Tree-of-Thoughts(ToT)、Cumulative Reasoning(CR)等)仍受到上下文扎根不足、幻觉或输出生成不一致,以及迭代迟缓等限制。为克服这些挑战,我们提出一种新颖的单步/双步零样本提示框架,该框架建立在两个独特策略之上:Chain of Evidences(CoE,证据链)和 Evidence to Generate(E2G,证据生成)。与未经证实的推理主张不同,我们的创新方法利用“证据用于决策”的力量:首先仅关注上下文中明确提到的思维序列,这些序列随后作为抽取的证据,以更高的精确性和效率指导 LLM 的输出生成过程。这一简单而有效的方法释放了思维链提示的全部潜力,促进 LLM 实现更快、更可靠且具备上下文感知能力的推理。我们的框架在各类知识密集型推理和生成任务上持续取得显著结果,并在使用最先进 LLM 时超越基线方法。例如:(i)在使用 GPT-4 的 LogiQA 基准上,CoE 达到 53.8% 的最新最先进准确率,分别比 CoT 高 18%、比 ToT 高 11%、比 CR 高 9%;(ii)结合 PaLM-2 的 CoE 在 DROP 上取得 83.3 的 F1 分数,比 Gemini Ultra 的可变样本性能高出 0.9 个 F1 点。我们将在这些基准上的提示和输出作为新的指令微调数据集发布,以供未来研究使用,地址为 https://huggingface.co/datasets/kagnlp/Chain-of-Evidences/。
英文摘要:
While chain-of-thoughts (CoT) prompting has revolutionized how LLMs perform reasoning tasks, its current methods and variations (e.g, Self-consistency, ReACT, Reflexion, Tree-of-Thoughts (ToT), Cumulative Reasoning (CR) etc.,) suffer from limitations like limited context grounding, hallucination/inconsistent output generation, and iterative sluggishness. To overcome these challenges, we introduce a novel mono/dual-step zero-shot prompting framework built upon two unique strategies Chain of Evidences (CoE)} and Evidence to Generate (E2G). Instead of unverified reasoning claims, our innovative approaches leverage the power of "evidence for decision making" by first focusing exclusively on the thought sequences explicitly mentioned in the context which then serve as extracted evidence, guiding the LLM's output generation process with greater precision and efficiency. This simple yet potent approach unlocks the full potential of chain-of-thoughts prompting, facilitating faster, more reliable, and contextually aware reasoning in LLMs. Our framework consistently achieves remarkable results across various knowledge-intensive reasoning and generation tasks, surpassing baseline approaches with state-of-the-art LLMs. For instance, (i) on the LogiQA benchmark using GPT-4, CoE achieves a new state-of-the-art accuracy of 53.8%, surpassing CoT by 18%, ToT by 11%, and CR by 9%; (ii) CoE with PaLM-2 outperforms the variable-shot performance of Gemini Ultra by 0.9 F1 points, achieving an F1 score of 83.3 on DROP. We release our prompts and outputs on these benchmarks as a new instruction tuning dataset for future research at https://huggingface.co/datasets/kagnlp/Chain-of-Evidences/.
Enhancing Document-level Argument Extraction with Definition-augmented Heuristic-driven Prompting for LLMs
School of Engineering and Informatics, University of Sussex(萨塞克斯大学工程与信息学院);International Business School Suzhou, Xi’an Jiaotong-Liverpool University(西交利物浦大学苏州国际商学院);Management School, University of Liverpool(利物浦大学管理学院)
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Chain of Grounded Objectives: Bridging Process and Goal-oriented Prompting for Code Generation
Electronics and Telecommunications Research Institute(电子电信研究院);Department of Computer Science and Engineering, Seoul National University(计算机科学与工程系,首尔国立大学)
UKP Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt(德国达姆施塔特技术大学UKP实验室、计算机科学系和海森人工智能中心(hessian.AI));Zuse School ELIZA(Zuse学院ELIZA);National Research Center for Applied Cybersecurity ATHENE(应用网络安全国家研究中心ATHENE);Indian Institute of Technology Delhi(印度德里理工学院);Yardi School of Artificial Intelligence(Yardi人工智能学院);University of Louisville(路易斯维尔大学);University of Toledo(托莱多大学)
Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine
University of Oxford(牛津大学);Peking University(北京大学);Yale University(耶鲁大学);Tencent(腾讯);Oxford-Suzhou Centre for Advanced Research(牛津-苏州高等研究院);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院)
Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations
Seoul National University(首尔国立大学);Human-Computer Interaction Institute, Carnegie Mellon University(人机交互研究所,卡内基梅隆大学);Language Technologies Institute, Carnegie Mellon University(语言技术研究所,卡内基梅隆大学)