POE:用于多项选择推理的排除法
POE: Process of Elimination for Multiple Choice Reasoning
AI总结:
提出二步排除法(POE),通过先评分排除错误选项再掩盖预测,提升语言模型在多项选择推理任务上的表现。
摘要:
语言模型(LMs)能够进行上下文学习以完成多项选择推理任务,但这些任务中的选项被同等对待。由于人类通常在选出最终正确答案前先排除错误选项,我们论证类似的二步策略能使LMs更好地完成这些任务。为此,我们提出排除法(POE),一种二步评分方法。第一步,POE对每个选项评分,并排除看似错误的选项。第二步,POE掩盖这些错误选项,并从剩余选项中做出最终预测。在8个推理任务上的零样本实验表明了POE的有效性,随后的分析发现我们的方法在逻辑推理任务上表现尤为出色。我们进一步分析了掩盖的效果,并表明POE适用于少样本设置以及如ChatGPT的大型语言模型(LLMs)。
英文摘要:
Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。