arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.15554cs.CL
收藏

“你指的是什么?”评估多模态对话模型处理澄清性交流的能力

'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges

Heriot-Watt University(赫瑞-瓦特大学);AlanaAI

Javier Chiyah-Garcia, Alessandro Suglia, Arash Eshghi, Helen Hastie

AI总结:

本文评估多模态对话模型处理澄清性交流的能力,发现语言模型擅长处理对话历史相关的澄清,而多模态模型通过额外学习目标获取解耦对象表示,以应对跨模态复杂指称歧义。

摘要:

当指称表达未能为听者唯一识别出预期所指对象时,对话中便会产生指称歧义。听者通常会立即察觉此类歧义,并通过元交际的澄清性交流(Clarificational Exchanges, CE)与说话者协作修复:即提出一个澄清请求(Clarification Request, CR)并获得回应。在此,我们认为生成和回应 CR 的能力对多模态视觉基础对话模型的架构和目标函数施加了特定约束。我们使用 SIMMC 2.0 数据集评估不同最先进模型架构处理 CE 的能力,并采用一种指标探测模型中由 CE 引发的上下文更新。我们发现,基于语言的模型能够编码简单的多模态语义信息并处理部分 CE,尤其擅长处理与对话历史相关的 CE;而多模态模型则能利用额外的学习目标获取解耦的对象表示,这对于整体上处理跨模态的复杂指称歧义至关重要。

英文摘要:

Referential ambiguities arise in dialogue when a referring expression does not uniquely identify the intended referent for the addressee. Addressees usually detect such ambiguities immediately and work with the speaker to repair it using meta-communicative, Clarificational Exchanges (CE): a Clarification Request (CR) and a response. Here, we argue that the ability to generate and respond to CRs imposes specific constraints on the architecture and objective functions of multi-modal, visually grounded dialogue models. We use the SIMMC 2.0 dataset to evaluate the ability of different state-of-the-art model architectures to process CEs, with a metric that probes the contextual updates that arise from them in the model. We find that language-based models are able to encode simple multi-modal semantic information and process some CEs, excelling with those related to the dialogue history, whilst multi-modal models can use additional learning objectives to obtain disentangled object representations, which become crucial to handle complex referential ambiguities across modalities overall.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑