认识你的错误:通过问责建模防止对任务导向对话AI的过度依赖
Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
AI总结:
提出一种用于基于LLM的任务导向对话智能体的问责模型,通过在模型不确定或DST出错时引入摩擦轮次,可靠估计错误并指导生成更准确操作,显著提升JGA并有效减少用户过度依赖。
摘要:
最近的 LLM 为对话智能体带来了显著的进步。然而,它们也因产生幻觉而众所周知,会生成看似合理但实际上不符合事实的回复。另一方面,用户往往过度依赖基于 LLM 的 AI 智能体,即使 AI 的建议是错误的,也会接受。添加正向摩擦(positive friction),例如提供解释或获取用户确认,已被提议作为 AI 辅助决策系统中的一种缓解措施。在本文中,我们提出了一种用于基于 LLM 的任务导向对话智能体的问责模型(accountability model),以在模型存在不确定性以及与对话状态跟踪(DST)相关的错误时,通过摩擦轮次(friction turns)来解决用户过度依赖的问题。该问责模型是一个增强型 LLM,带有一个额外的问责头(accountability head),该头作为一个二元分类器,用于预测对话中提到的对话状态的相关槽位。我们在两个已建立的基准(MultiWOZ 和 Snips)上使用多个骨干 LLM 进行了实验。我们的实证结果表明,所提出的方法不仅能够可靠地估计 AI 智能体的错误,还能指导解码器生成更准确的操作。我们观察到,通过将问责头纳入现代 LLM 中,DST 输出的联合目标准确率(JGA)获得了约 3% 的绝对提升。对检测到的错误进行自我纠正进一步将 JGA 从 67.13 提升至 70.51,达到了最先进的 DST 性能。最后,我们展示了通过用户确认(摩擦轮次)进行错误纠正也能获得相似的性能提升,凸显了其在减少用户过度依赖方面的潜力。
英文摘要:
Recent LLMs have enabled significant advancements for conversational agents. However, they are also well known to hallucinate, producing responses that seem plausible but are factually incorrect. On the other hand, users tend to over-rely on LLM-based AI agents, accepting AI's suggestion even when it is wrong. Adding positive friction, such as explanations or getting user confirmations, has been proposed as a mitigation in AI-supported decision-making systems. In this paper, we propose an accountability model for LLM-based task-oriented dialogue agents to address user overreliance via friction turns in cases of model uncertainty and errors associated with dialogue state tracking (DST). The accountability model is an augmented LLM with an additional accountability head that functions as a binary classifier to predict the relevant slots of the dialogue state mentioned in the conversation. We perform our experiments with multiple backbone LLMs on two established benchmarks (MultiWOZ and Snips). Our empirical findings demonstrate that the proposed approach not only enables reliable estimation of AI agent errors but also guides the decoder in generating more accurate actions. We observe around 3% absolute improvement in joint goal accuracy (JGA) of DST output by incorporating accountability heads into modern LLMs. Self-correcting the detected errors further increases the JGA from 67.13 to 70.51, achieving state-of-the-art DST performance. Finally, we show that error correction through user confirmations (friction turn) achieves a similar performance gain, highlighting its potential to reduce user overreliance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。