arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.00602cs.CL
收藏

使用文本和图像的兴趣点类型预测

Point-of-Interest Type Prediction using Text and Images

University of Sheffield(谢菲尔德大学)

Danae Sánchez Villegas, Nikolaos Aletras

摘要:

兴趣点(POI)类型预测是推断社交媒体帖子被共享的地方的类型的任务。推断POI的类型对于包括社会语言学、地理符号学和文化地理学在内的计算社会科学的研究是有用的,并且在诸如推荐和可视化系统的地理社会网络技术中具有应用。POI类型预测中的先前努力仅关注文本,而不考虑视觉信息。然而,在现实中,形式的多样性,以及它们之间的符号关系,塑造了社交媒体中的沟通和互动。本文提出了一种研究POI类型预测使用多模态信息,从文本和图像在发布时间。为此,我们丰富了目前可用的数据集POI类型预测与伴随的文本消息的图像。我们提出的方法从每个模态中提取相关信息,以有效地捕获文本和图像之间的交互,在八个类别中实现47.21的宏F1,显着优于基于纯文本方法的POI类型预测的最先进方法。最后,我们提供了一个详细的分析,阐明跨模态的相互作用和我们的最佳性能模型的局限性。

英文摘要:

Point-of-interest (POI) type prediction is the task of inferring the type of a place from where a social media post was shared. Inferring a POI's type is useful for studies in computational social science including sociolinguistics, geosemiotics, and cultural geography, and has applications in geosocial networking technologies such as recommendation and visualization systems. Prior efforts in POI type prediction focus solely on text, without taking visual information into account. However in reality, the variety of modalities, as well as their semiotic relationships with one another, shape communication and interactions in social media. This paper presents a study on POI type prediction using multimodal information from text and images available at posting time. For that purpose, we enrich a currently available data set for POI type prediction with the images that accompany the text messages. Our proposed method extracts relevant information from each modality to effectively capture interactions between text and image achieving a macro F1 of 47.21 across eight categories significantly outperforming the state-of-the-art method for POI type prediction based on text-only methods. Finally, we provide a detailed analysis to shed light on cross-modal interactions and the limitations of our best performing model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑