MMWOZ：构建面向任务型对话的多模态智能体
MMWOZ: Building Multimodal Agent for Task-oriented Dialogue
AI总结：
针对传统任务型对话系统难以适配无定制后端API的前端GUI场景的问题，本文构建多模态对话数据集MMWOZ，提出基线模型MATE并开展实验分析。
摘要：
面向任务型对话系统因具备完成用户目标（如为用户预订机票）的对话能力而受到广泛关注。传统上，这类系统被定义为使用自然语言与用户交互、可访问定制化后端API的智能体，但在真实场景中，前端图形用户界面（GUI）普遍存在，同时缺少定制化后端API，导致传统面向任务型对话系统在实际应用中存在显著短板。为填补这一差距，本文构建了从MultiWOZ 2.3数据集扩展而来的新型多模态对话数据集MMWOZ：首先开发网页风格GUI作为前端，其次设计自动化脚本将原数据集的对话状态与系统动作转换为GUI操作指令，最后采集网页快照及其对应的操作指令。此外，本文提出名为MATE（面向任务型对话的多模态智能体，Multimodal Agent for Task-oriEnted dialogue）的新型多模态模型作为MMWOZ数据集的基线模型，并基于MATE开展全面实验分析，探究实用型面向任务型对话多模态智能体的构建方案。
英文摘要：
Task-oriented dialogue systems have garnered significant attention due to their conversational ability to accomplish goals, such as booking airline tickets for users. Traditionally, task-oriented dialogue systems are conceptualized as intelligent agents that interact with users using natural language and have access to customized back-end APIs. However, in real-world scenarios, the widespread presence of front-end Graphical User Interfaces (GUIs) and the absence of customized back-end APIs create a significant gap for traditional task-oriented dialogue systems in practical applications. In this paper, to bridge the gap, we collect MMWOZ, a new multimodal dialogue dataset that is extended from MultiWOZ 2.3 dataset. Specifically, we begin by developing a web-style GUI to serve as the front-end. Next, we devise an automated script to convert the dialogue states and system actions from the original dataset into operation instructions for the GUI. Lastly, we collect snapshots of the web pages along with their corresponding operation instructions. In addition, we propose a novel multimodal model called MATE (Multimodal Agent for Task-oriEnted dialogue) as the baseline model for the MMWOZ dataset. Furthermore, we conduct comprehensive experimental analysis using MATE to investigate the construction of a practical multimodal agent for task-oriented dialogue.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
