DialSim：用于评估对话智能体长期多方对话理解能力的对话模拟器
DialSim: A Dialogue Simulator for Evaluating Long-Term Multi-Party Dialogue Understanding of Conversational Agents
AI总结：
针对现有对话智能体评估忽视多方对话与长期依赖的问题，提出DialSim模拟评估框架及LongDialQA数据集，发现大上下文或RAG模型仍难以保持长期多方理解，凸显更真实基准的必要性。
摘要：
大型语言模型（LLM）的最新进展显著增强了对话智能体，使其可应用于教育、娱乐等多个领域。尽管取得了进展，但对智能体的评估往往忽视了真实世界对话的复杂性，例如多方对话和长期上下文依赖。为弥补这一差距，我们提出了DialSim，一个基于对话模拟的评估框架。在DialSim中，智能体扮演脚本化对话中的某个角色，并仅依据对话历史来回答自发问题，同时需识别自身信息不足的情况。为支持该框架，我们引入了LongDialQA，一个从长篇电视剧中构建的新问答数据集，包含超过1,300个对话会话，每个会话配有超过1,000个精心设计的问题，总计超过352,000个词元。为尽量减少对先验知识的依赖，所有角色名称均被匿名化或替换。我们使用DialSim对基于LLM的最先进对话智能体进行评估，结果显示，即使具有大上下文窗口或RAG能力的模型也难以在长期多方交互中保持准确理解，这凸显了对话AI领域对更真实且更具挑战性基准的需求。
英文摘要：
Recent advancements in Large Language Models (LLMs) have significantly enhanced conversational agents, making them applicable to various fields (e.g., education, entertainment). Despite their progress, the evaluation of the agents often overlooks the complexities of real-world conversations, such as multi-party dialogues and extended contextual dependencies. To bridge this gap, we introduce DialSim, a dialogue simulation-based evaluation framework. In DialSim, an agent assumes the role of a character in a scripted conversation and is evaluated on their ability to answer spontaneous questions using only the dialogue history, while recognizing when they lack sufficient information. To support this framework, we introduce LongDialQA, a new QA dataset constructed from long-running TV shows, comprising over 1,300 dialogue sessions, each paired with more than 1,000 carefully curated questions, totaling over 352,000 tokens. To minimize reliance on prior knowledge, all character names are anonymized or swapped. Our evaluation of state-of-the-art LLM-based conversational agents using DialSim reveals that even models with large context windows or RAG capabilities struggle to maintain accurate comprehension over long-term, multi-party interactions-underscoring the need for more realistic and challenging benchmarks in conversational AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
