arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.07532cs.CL
收藏

犯错并非易事:大型语言模型在排除法推理中面临困难

It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning

University of Maryland(马里兰大学)

Nishant Balepur, Shramay Palta, Rachel Rudinger

AI总结:

本文研究大型语言模型在多项选择题中使用排除法推理错误选项的能力,发现排除法策略始终不如直接选择正确答案,且两种策略间一致性低于各自自一致性,并通过错误分析为未来研究提供建议。

摘要:

思维链(COT)提示可以帮助大型语言模型(LLMs)推理出正确答案,但其在推理出错误答案方面的有效性尚未被探索。这种排除法(PoE)在与COT结合使用时,可以增强自一致性、可解释性,并有助于排除性医学诊断等任务。因此,我们提出了PoE与COT的结合方法,即要求LLMs在多项选择题上对错误选项进行推理。我们在总共四个常识推理和科学推理数据集上评估了GPT-3.5、LLaMA-2和Falcon执行PoE与COT的能力。我们发现,排除法策略的表现始终不如选择正确答案的策略。这两种策略之间的一致性也低于每种策略自身的自一致性。为了进一步研究这些问题,我们进行了错误分析,并为未来工作提出了建议。

英文摘要:

Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑