评估大型语言模型在信息寻求对话中基于文档的响应生成
Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
AI总结：
本文评估ChatGPT等大型语言模型在信息寻求对话中基于文档生成响应的能力，通过Chat-Completion和LlamaIndex两种方法在MultiDoc2Dial语料库上进行人工评估，发现ChatGPT变体虽更易产生幻觉但评分仍高于共享任务获胜系统和人类响应。
摘要：
在本文中，我们研究了在信息寻求对话场景下使用ChatGPT等大型语言模型（LLMs）进行基于文档的响应生成。为了进行评估，我们使用MultiDoc2Dial语料库，该语料库包含四个社会服务领域的任务导向对话，此前曾在DialDoc 2022共享任务中使用。信息寻求对话轮次以多个提供相关信息的文档为基础。我们通过提示ChatGPT模型来生成对话补全响应，采用两种方法：Chat-Completion和LlamaIndex。ChatCompletion利用ChatGPT模型预训练中的知识，而LlamaIndex还从文档中提取相关信息。我们观察到，由于LLMs生成的响应明显更加冗长，基于文档的响应生成无法通过自动评估指标得到充分评估，因此我们进行了人工评估，由标注者对共享任务获胜系统的输出、两种ChatGPT变体的输出以及人类响应进行评分。虽然两种ChatGPT变体更可能包含相关片段中不存在的信息，可能包括幻觉的存在，但它们的评分高于共享任务获胜系统和人类响应。
英文摘要：
In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus of task-oriented dialogues in four social service domains previously used in the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded in multiple documents providing relevant information. We generate dialogue completion responses by prompting a ChatGPT model, using two methods: Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT model pretraining while LlamaIndex also extracts relevant information from documents. Observing that document-grounded response generation via LLMs cannot be adequately assessed by automatic evaluation metrics as they are significantly more verbose, we perform a human evaluation where annotators rate the output of the shared task winning system, the two Chat-GPT variants outputs, and human responses. While both ChatGPT variants are more likely to include information not present in the relevant segments, possibly including a presence of hallucinations, they are rated higher than both the shared task winning system and human responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
