arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.06729cs.CLcs.AI
收藏

STREET:一个多任务结构化推理和解释基准

STREET: A Multi-Task Structured Reasoning and Explanation Benchmark

AWS AI(AWS人工智能研究院);Northwestern University(西北大学)

Danilo Ribeiro, Shen Wang, Xiaofei Ma, Henry Zhu, Rui Dong, Deguang Kong, Juliette Burger, Anjelica Ramos, William Wang, Zhiheng Huang, George Karypis, Bing Xiang, Dan Roth

摘要:

我们介绍STREET,一个统一的多任务和多领域的自然语言推理和解释基准。与大多数现有的问答(QA)数据集不同,我们期望模型不仅回答问题,而且还产生逐步的结构化解释,描述如何使用问题中的前提来产生可以证明某个答案正确性的中间结论。我们使用流行的语言模型进行广泛的评估,例如Few-Shot提示GPT-3和微调的T5。我们发现,这些模型仍然落后于人类的表现时,产生这种结构化的推理步骤。我们相信这项工作将为社区提供一种方法,以更好地训练和测试多步推理和自然语言解释系统。

英文摘要:

We introduce STREET, a unified multi-task and multi-domain natural language reasoning and explanation benchmark. Unlike most existing question-answering (QA) datasets, we expect models to not only answer questions, but also produce step-by-step structured explanations describing how premises in the question are used to produce intermediate conclusions that can prove the correctness of a certain answer. We perform extensive evaluation with popular language models such as few-shot prompting GPT-3 and fine-tuned T5. We find that these models still lag behind human performance when producing such structured reasoning steps. We believe this work will provide a way for the community to better train and test systems on multi-step reasoning and explanations in natural language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑