TREC iKAT 2023：用于评估对话式与交互式知识助手的测试集
TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants
AI总结：
TREC iKAT 2023构建了包含36个个性化对话、20个主题和PTKB的测试集，通过344轮对话及多维度评估，为对话式搜索代理研究提供基准。
摘要：
随着大语言模型（LLMs）的发展，对话式信息寻求在过去几年中迅速演进，为以自然方式解释和响应用户请求提供了基础。扩展的TREC交互式知识辅助赛道（iKAT）集合旨在使研究人员能够测试和评估他们的对话式搜索代理（CSA）。该集合包含36个个性化对话，涵盖20个不同主题，每个对话都配有一个个人文本知识库（PTKB），用于定义定制的用户画像。共提供344轮对话，约26,000个段落，作为相关性评估，同时还对生成的响应在四个关键维度上进行额外评估：相关性、完整性、有据性和自然性。该集合挑战CSA高效导航多样化的个人上下文，引出相关的画像信息，并利用上下文进行相关对话。PTKB的整合以及对决策性搜索任务的强调，促成了该测试集的独特性，使其成为推进对话式和交互式知识助手研究的重要基准。
英文摘要：
Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
