ChatGPT能否取代传统KBQA模型？对GPT LLM家族问答性能的深入分析
Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family
AI总结：
本文通过CheckList黑盒测试框架，在约19万个测试用例上系统评估GPT LLM家族在八个复杂知识问答数据集上的表现，并与FLAN-T5对比，分析其能否取代传统KBQA模型。
摘要：
ChatGPT是一种强大的大语言模型（LLM），其覆盖了维基百科等知识资源，并支持利用自身知识进行自然语言问答。因此，人们越来越关注探索ChatGPT能否取代传统的基于知识的问答（KBQA）模型。尽管已有一些工作分析了ChatGPT的问答性能，但仍缺乏对各种类型复杂问题的大规模、全面测试，以分析该模型的局限性。在本文中，我们提出了一个遵循Ribeiro等人提出的CheckList黑盒测试规范的框架。我们在八个真实世界的基于知识的复杂问答数据集上评估了ChatGPT及其LLM家族，其中包括六个英文数据集和两个多语言数据集。测试用例总数约为190,000个。除了GPT家族的LLM之外，我们还评估了知名的FLAN-T5，以识别GPT家族与其他LLM之间的共性。数据集和代码可在https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family.git获取。
英文摘要：
ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge. Therefore, there is growing interest in exploring whether ChatGPT can replace traditional knowledge-based question answering (KBQA) models. Although there have been some works analyzing the question answering performance of ChatGPT, there is still a lack of large-scale, comprehensive testing of various types of complex questions to analyze the limitations of the model. In this paper, we present a framework that follows the black-box testing specifications of CheckList proposed by Ribeiro et. al. We evaluate ChatGPT and its family of LLMs on eight real-world KB-based complex question answering datasets, which include six English datasets and two multilingual datasets. The total number of test cases is approximately 190,000. In addition to the GPT family of LLMs, we also evaluate the well-known FLAN-T5 to identify commonalities between the GPT family and other LLMs. The dataset and code are available at https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family.git
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
