Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog
WICT, Peking University(北京大学王选计算机研究所);Inception Institute of Artificial Intelligence(Inception人工智能研究所);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Shen Gao, Xiuying Chen, Chang Liu, Li Liu, Dongyan Zhao, Rui Yan
Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching text labels of stickers with previous utterances. However, due to their large quantities, it is impractical to require text labels for the all stickers. Hence, in this paper, we propose to recommend an appropriate sticker to user based on multi-turn dialog context history without any external labels. Two main challenges are confronted in this task. One is to learn semantic meaning of stickers without corresponding text labels. Another challenge is to jointly model the candidate sticker with the multi-turn dialog context. To tackle these challenges, we propose a sticker response selector (SRS) model. Specifically, SRS first employs a convolutional based sticker image encoder and a self-attention based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker with each utterance in the dialog history. SRS then learns the short-term and long-term dependency between all interaction results by a fusion network to output the the final matching score. To evaluate our proposed method, we collect a large-scale real-world dialog dataset with stickers from one of the most popular online chatting platform. Extensive experiments conducted on this dataset show that our model achieves the state-of-the-art performance for all commonly-used metrics. Experiments also verify the effectiveness of each component of SRS. To facilitate further research in sticker selection field, we release this dataset of 340K multi-turn dialog and sticker pairs.
Teaching Text-to-Image Models to Communicate in Dialog
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);National Key Laboratory of General Artificial Intelligence, BIGAI(北京智源人工智能研究院通用人工智能全国重点实验室);Department of Computer Science, The Open University of China(国家开放大学计算机系)
ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue
PAII Inc.(PAII公司);Ping An Technology(平安科技);Ping An Healthcare and Technology Company Limited(平安医疗健康科技有限公司);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Tianjin University(天津大学)
Facilitating Multi-turn Emotional Support Conversation with Positive Emotion Elicitation: A Reinforcement Learning Approach
Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学北京信息科学与技术国家研究中心);Tsinghua University(清华大学);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Tianjin University(天津大学)