DialogBench：评估大语言模型作为类人对话系统
DialogBench: Evaluating LLMs as Human-like Dialogue Systems
AI总结：
本文提出包含12个对话任务的评估基准DialogBench，通过GPT-4生成高质量评估实例，并在26个中英文大语言模型上测试，发现指令微调虽提升类人程度但仍不足，且助手定位可能削弱情感感知与日常生活知识掌握。
摘要：
大语言模型（LLMs）通过利用指令微调在新的对话能力方面取得了显著突破，这刷新了人们对对话系统的印象。对话系统的长期目标是足够类人化，以便与用户建立长期联系。因此，迫切需要将LLMs作为类人对话系统进行评估。在本文中，我们提出了DialogBench，一个包含12个对话任务的对话评估基准，用于探测LLMs作为类人对话系统应具备的能力。具体来说，我们提示GPT-4为每个任务生成评估实例。我们首先基于广泛使用的设计原则设计基础提示，并进一步缓解现有偏差以生成更高质量的评估实例。我们对26个LLMs在英文和中文DialogBench上的广泛测试表明，指令微调在一定程度上提高了LLMs的类人程度，但大多数LLMs作为类人对话系统仍有很大的改进空间。有趣的是，结果还表明，助手AI的定位可能使指令微调削弱LLMs的人类情感感知能力及其对人类日常生活信息的掌握。
英文摘要：
Large language models (LLMs) have achieved remarkable breakthroughs in new dialogue capabilities by leveraging instruction tuning, which refreshes human impressions of dialogue systems. The long-standing goal of dialogue systems is to be human-like enough to establish long-term connections with users. Therefore, there has been an urgent need to evaluate LLMs as human-like dialogue systems. In this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have. Specifically, we prompt GPT-4 to generate evaluation instances for each task. We first design the basic prompt based on widely used design principles and further mitigate the existing biases to generate higher-quality evaluation instances. Our extensive tests on English and Chinese DialogBench of 26 LLMs show that instruction tuning improves the human likeness of LLMs to a certain extent, but most LLMs still have much room for improvement as human-like dialogue systems. Interestingly, results also show that the positioning of assistant AI can make instruction tuning weaken the human emotional perception of LLMs and their mastery of information about human daily life.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
