重新思考对话系统的评估：用户反馈对众包工作者和大型语言模型的影响
Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
AI总结：
本文研究用户后续话语反馈对任务导向对话系统评估的影响，通过众包工作者和大型语言模型在四种维度上的对比实验，发现用户反馈显著影响评分，且工作者更易受反馈影响，为自动反馈集成评估提供了依据。
摘要：
在即席检索中，评估在很大程度上依赖于用户行为，包括隐式反馈。在对话场景中，由于交互的性质，此类信号通常不可用，因此评估往往依赖于众包评估标签。用户反馈在标注者评估对话感知中的轮次时所起的作用很少被研究。我们重点关注任务导向对话系统（TDSs）的评估如何受到用户反馈（显式或隐式）的影响，这些反馈通过被评估轮次的后续话语提供。我们探索并比较了两种评估 TDS 的方法：一种包含用户的后续话语，另一种不包含。我们同时使用众包工作者和大型语言模型（LLMs）作为标注者，从四个维度评估系统响应：相关性、有用性、趣味性和解释质量。我们的研究结果表明，在两种设置中，两组标注者给出的评分存在明显差异，这表明用户反馈确实会影响系统评估。与 LLMs 在趣味性和相关性上相比，工作者在有用性和趣味性上更容易受到用户反馈的影响。用户反馈使工作者对有用性的评估更加个性化，与用户的显式反馈高度一致。此外，在用户请求模糊或复杂的情况下，用户反馈提高了众包工作者之间的一致性。这些发现强调了用户反馈在改进系统评估中的重要性，并表明未来研究中自动反馈集成的潜力。我们公开发布标注数据，以促进该领域的研究。
英文摘要：
In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. In a conversational setting such signals are usually unavailable due to the nature of the interactions, and, instead, the evaluation often relies on crowdsourced evaluation labels. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems (TDSs), is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated. We explore and compare two methodologies for assessing TDSs: one includes the user's follow-up utterance and one without. We use both crowdworkers and large language models (LLMs) as annotators to assess system responses across four aspects: relevance, usefulness, interestingness, and explanation quality. Our findings indicate that there is a distinct difference in ratings assigned by both annotator groups in the two setups, indicating user feedback does influence system evaluation. Workers are more susceptible to user feedback on usefulness and interestingness compared to LLMs on interestingness and relevance. User feedback leads to a more personalized assessment of usefulness by workers, aligning closely with the user's explicit feedback. Additionally, in cases of ambiguous or complex user requests, user feedback improves agreement among crowdworkers. These findings emphasize the significance of user feedback in refining system evaluations and suggest the potential for automated feedback integration in future research. We publicly release the annotated data to foster research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
