arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.13332cs.CL
收藏

RealTime QA:此刻的答案是什么?

RealTime QA: What's the Answer Right Now?

Toyota Technological Institute at Chicago(芝加哥丰田理工学院);Tohoku University(东北大学);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学);Yale University(耶鲁大学);RIKEN(理化学研究所);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, Kentaro Inui

AI总结:

本文提出按周更新的动态开放域问答基准 REALTIME QA,并基于 GPT-3、T5 评估系统回答最新事件的能力,揭示实时检索及识别不可答问题的重要性。

摘要:

我们介绍 REALTIME QA,这是一个动态问答(QA)平台,会定期发布问题并评估系统(在此版本中为每周一次)。REALTIME QA 询问当前世界的情况,问答系统需要回答有关新事件或新信息的问题。因此,它挑战了开放域问答数据集中静态的、传统的假设,并追求即时应用。我们在大型预训练语言模型(包括 GPT-3 和 T5)之上构建了强大的基线模型。我们的基准是一项持续推进的工作,本文呈现了过去一年的实时评估结果。实验结果表明,GPT-3 通常能够根据新检索到的文档恰当地更新其生成结果,这凸显了最新信息检索的重要性。尽管如此,我们发现当检索到的文档没有提供足以找到答案的信息时,GPT-3 往往会返回过时的答案。这指出了未来研究的一个重要方向:开放域问答系统能否识别此类无法回答的情况,并与用户甚至检索模块沟通,以修改检索结果?我们希望 REALTIME QA 将推动问答及其之外的即时应用取得进展。

英文摘要:

We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑