车载对话助手中多轮对话的自动化评估
Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
浏览论文内容
中文总结 AI 辅助
针对车载对话助手多轮交互评估难题，提出基于策略引导用户模拟器与双层LLM评判器的自动化框架，显著提升失败类型与失败对话的发现效率。
中文摘要 AI 辅助
车载对话助手（ICAs）日益集成到车辆中，以支持路线规划、车辆控制和信息访问。由于多轮交互、缺乏明确的地面真值以及严格的安全约束，确保其可靠性具有挑战性。现有评估技术存在不足，因为它们针对单轮设置，无法捕捉跨轮次的约束处理、上下文保留和安全关键行为。我们提出了一种自动化框架，用于测试ICA的多轮对话能力。该系统被视为黑盒，并通过闭环模拟进行评估，该模拟使用策略引导的用户模拟器、对抗性策略管理器和两层LLM评判器，分别评估轮次级失败和对话级质量。我们在具有六个LLM后端和十二名人工标注者的工业ICA上评估了该方法。自动化评判器与人类表现出显著一致性，策略引导相比无引导模拟，每段对话发现的独特失败类型多出2.96倍，独特失败对话数量增加了一倍以上。
英文摘要
In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. Ensuring their reliability is challenging due to multi-turn interactions, the absence of explicit ground truth, and strict safety constraints. Existing evaluation techniques fall short, as they target single-turn settings and fail to capture constraint handling, context retention, and safety-critical behavior across turns. We propose an automated framework for testing the multi-turn conversational capabilities of ICAs. The system is treated as a black box and evaluated via closed-loop simulation with a strategy-guided user simulator, an adversarial strategy manager, and a two-tier LLM judge assessing turn-level failures and conversation-level quality. We evaluate the approach on an industrial ICA with six LLM backends and twelve human annotators. The automated judge shows substantial agreement with humans, and strategy guidance uncovers 2.96 times more unique failure types per conversation and more than doubles the number of unique failing conversations compared to unguided simulation.
发表机构
- Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
- BMW Group（宝马集团）
- Technical University of Munich（慕尼黑工业大学）
- RWTH Aachen University（亚琛工业大学）
- fortiss GmbH（fortiss有限公司）
机构由 AI 辅助整理，请以论文原文为准。
