arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.03322cs.CLcs.CVcs.MM
收藏

学会用你最喜欢的贴纸回应:一个统一多轮对话中多模态和用户偏好的框架

Learning to Respond with Your Favorite Stickers: A Framework of Unifying Multi-Modality and User Preference in Multi-Turn Dialog

Peking University(北京大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);Renmin University of China(中国人民大学)

Shen Gao, Xiuying Chen, Li Liu, Dongyan Zhao, Rui Yan

摘要:

具有生动和吸引人的表情的贴纸在在线消息应用中变得越来越受欢迎,并且一些作品致力于通过将贴纸图像与先前的话语进行匹配来自动选择贴纸响应。然而,现有的方法通常侧重于测量对话上下文和贴纸图像之间的匹配度,这忽略了用户使用贴纸的偏好。因此,在本文中,我们建议推荐一个合适的贴纸给用户的基础上,多轮对话上下文和贴纸使用历史的用户。这项任务面临两个主要挑战。一种是基于用户先前的标签选择历史来建模标签偏好。另一个挑战是将用户偏好和对话上下文与候选贴纸之间的匹配联合融合到最终的预测中。为了应对这些挑战,我们提出了一个\emph{Preference Enhanced Sticker Response Selector}(PESRS)模型。具体而言,PESRS首先采用基于卷积的贴纸图像编码器和基于自我注意的多轮对话编码器来获得贴纸和话语的表示。接下来,提出了深度交互网络来进行贴纸与每个话语之间的深度匹配。然后,我们通过使用最近选择的贴纸作为输入来建模用户偏好,并使用键值记忆网络来存储偏好表示。PESRS通过融合网络学习所有交互结果之间的短期和长期依赖关系,并将用户偏好表示动态融合到最终的贴纸选择预测中。在大规模真实对话数据集上进行的大量实验表明,我们的模型在所有常用指标上都达到了最先进的性能。实验也验证了PESRS各组成部分的有效性。

英文摘要:

Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching the stickers image with previous utterances. However, existing methods usually focus on measuring the matching degree between the dialog context and sticker image, which ignores the user preference of using stickers. Hence, in this paper, we propose to recommend an appropriate sticker to user based on multi-turn dialog context and sticker using history of user. Two main challenges are confronted in this task. One is to model the sticker preference of user based on the previous sticker selection history. Another challenge is to jointly fuse the user preference and the matching between dialog context and candidate sticker into final prediction making. To tackle these challenges, we propose a \emph{Preference Enhanced Sticker Response Selector} (PESRS) model. Specifically, PESRS first employs a convolutional based sticker image encoder and a self-attention based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker and each utterance. Then, we model the user preference by using the recently selected stickers as input, and use a key-value memory network to store the preference representation. PESRS then learns the short-term and long-term dependency between all interaction results by a fusion network, and dynamically fuse the user preference representation into the final sticker selection prediction. Extensive experiments conducted on a large-scale real-world dialog dataset show that our model achieves the state-of-the-art performance for all commonly-used metrics. Experiments also verify the effectiveness of each component of PESRS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑