arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.07935cs.CLcs.AIcs.LG
收藏

面向社交帖子位置推断的多模态表示学习

Multi-modal Representation Learning for Social Post Location Inference

University of Electronic Science and Technology of China(电子科技大学);Swinburne University of Technology(斯威本科技大学)

Ruiting Dai, Jiayi Luo, Xucheng Luo, Lisi Mo, Wanlun Ma, Fan Zhou

AI总结:

针对社交帖子位置推断中多模态信息协同作用受关注不足的问题,提出融合多模态信息的MRLF框架,通过多头注意力和字符感知模块提升推断精度,为相关在线任务提供新思路。

摘要:

通过社交帖子推断地理位置对诸多实用的基于位置的应用至关重要,例如产品营销、兴趣点推荐以及新型冠状病毒肺炎感染者追踪。与基于图像的位置检索或基于社交帖子文本嵌入的位置推断不同,多模态信息(即帖子图像、文本和话题标签)在社交帖子定位中的协同作用受到的关注较少。本研究中,我们从Instagram收集了包含图像、文本和话题标签的真实社交帖子数据集,并提出了一种新颖的多模态表示学习框架(Multi-modal Representation Learning Framework,MRLF),该框架能够融合社交帖子的不同模态以进行位置推断。MRLF集成了多头注意力机制,以增强位置显著信息的提取,同时与基于单域的方法相比,显著提升了位置推断效果。为了克服用户生成文本内容中的噪声问题,我们引入了一种新颖的基于注意力的字符感知模块,该模块考虑社交帖子文本和话题标签的字符之间的相对依赖关系,以实现灵活的多模态信息融合。实验结果表明,MRLF能够进行准确的位置预测,并为理解社交帖子多模态数据以用于在线推断任务开辟了新的途径。

英文摘要:

Inferring geographic locations via social posts is essential for many practical location-based applications such as product marketing, point-of-interest recommendation, and infector tracking for COVID-19. Unlike image-based location retrieval or social-post text embedding-based location inference, the combined effect of multi-modal information (i.e., post images, text, and hashtags) for social post positioning receives less attention. In this work, we collect real datasets of social posts with images, texts, and hashtags from Instagram and propose a novel Multi-modal Representation Learning Framework (MRLF) capable of fusing different modalities of social posts for location inference. MRLF integrates a multi-head attention mechanism to enhance location-salient information extraction while significantly improving location inference compared with single domain-based methods. To overcome the noisy user-generated textual content, we introduce a novel attention-based character-aware module that considers the relative dependencies between characters of social post texts and hashtags for flexible multi-model information fusion. The experimental results show that MRLF can make accurate location predictions and open a new door to understanding the multi-modal data of social posts for online inference tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑