arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.12692cs.MMcs.CV
收藏

面向多模态文本和图像数据的情感反馈合成

Affective Feedback Synthesis Towards Multimodal Text and Image Data

Indian Institute of Technology Roorkee(印度理工学院鲁尔基分校);Indian Institute of Technology Hyderabad(印度理工学院海得拉巴分校);National Institute of Technology Karnataka(卡纳塔克邦国家理工学院)

Puneet Kumar, Gaurav Bhat, Omkar Ingle, Daksh Goyal, Balasubramanian Raman

摘要:

在本文中,我们已经定义了一个新的任务,情感反馈合成,处理输入文本和相应的图像以类似的方式,人类对多模态数据的反应产生反馈。一个反馈合成系统已经提出并训练使用地面实况人类评论以及图像-文本输入。我们还构建了一个大规模的数据集,包括图像,文本,Twitter用户评论,以及通过Twitter提要抓取新闻文章的评论点赞数。所提出的系统使用基于变换器的文本编码器提取文本特征,而视觉特征已经使用更快的基于区域的卷积神经网络模型提取。文本和视觉特征已经被连接以构建多模态特征,解码器使用该多模态特征来合成反馈。我们比较了所提出的系统的结果与基线模型使用定量和定性的措施。生成的反馈进行了分析,使用自动和人工评价。它们被发现在语义上类似于地面实况评论,并且与给定的文本图像输入相关。

英文摘要:

In this paper, we have defined a novel task of affective feedback synthesis that deals with generating feedback for input text & corresponding image in a similar way as humans respond towards the multimodal data. A feedback synthesis system has been proposed and trained using ground-truth human comments along with image-text input. We have also constructed a large-scale dataset consisting of image, text, Twitter user comments, and the number of likes for the comments by crawling the news articles through Twitter feeds. The proposed system extracts textual features using a transformer-based textual encoder while the visual features have been extracted using a Faster region-based convolutional neural networks model. The textual and visual features have been concatenated to construct the multimodal features using which the decoder synthesizes the feedback. We have compared the results of the proposed system with the baseline models using quantitative and qualitative measures. The generated feedbacks have been analyzed using automatic and human evaluation. They have been found to be semantically similar to the ground-truth comments and relevant to the given text-image input.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑