语境至关重要：面向任务的对话系统中众包评估标签的影响
Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
AI总结：
该研究探究对话语境对面向任务对话系统众包标注质量的影响，发现截断语境会使评分偏正面，完整语境提升相关性标注质量但增加有用性标注歧义，用首轮用户话语作语境可兼顾标注一致性与效率。
摘要：
众包标签在评估面向任务的对话系统（TDSs）中发挥着关键作用。从标注者处获取高质量且一致的真实标签存在挑战。评估TDS时，标注者必须充分理解对话内容后才能给出判断。此前的研究提出在标注过程中仅使用部分对话语境，但这种限制对标签质量的影响尚未得到探究。本研究考察了对话语境对标注质量的影响，针对相关性和有用性标注任务，研究了截断语境的作用。我们进一步提出使用大语言模型（LLMs）对对话语境进行总结，以生成内容丰富且简洁的对话语境描述，并研究这种做法对标注者表现的影响。减少语境会导致更多的正面评分。反之，提供完整对话语境能带来更高质量的相关性评分，但会在有用性评分中引入歧义。将用户的首轮话语作为语境，能得到与使用完整对话相近的一致评分，且大幅降低了标注工作量。我们的研究结果揭示了任务设计，尤其是对话语境的可获得性，如何影响众包评估标签的质量与一致性。
英文摘要：
Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems (TDSs). Obtaining high-quality and consistent ground-truth labels from annotators presents challenges. When evaluating a TDS, annotators must fully comprehend the dialogue before providing judgments. Previous studies suggest using only a portion of the dialogue context in the annotation process. However, the impact of this limitation on label quality remains unexplored. This study investigates the influence of dialogue context on annotation quality, considering the truncated context for relevance and usefulness labeling. We further propose to use large language models (LLMs) to summarize the dialogue context to provide a rich and short description of the dialogue context and study the impact of doing so on the annotator's performance. Reducing context leads to more positive ratings. Conversely, providing the entire dialogue context yields higher-quality relevance ratings but introduces ambiguity in usefulness ratings. Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort. Our findings show how task design, particularly the availability of dialogue context, affects the quality and consistency of crowdsourced evaluation labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
