arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.10079cs.CVcs.AI
收藏

M3PT:一种用于POI标注的多模态模型

M3PT: A Multi-Modal Model for POI Tagging

Fudan University(复旦大学);Alibaba Group(阿里巴巴集团);East China University of Science and Technology(华东理工大学)

Jingsong Yang, Guanzhou Han, Deqing Yang, Jingping Liu, Yanghua Xiao, Xiang Xu, Baohua Wu, Shenghua Ni

AI总结:

本文提出多模态模型M3PT,融合POI文本与视觉特征并采用对比学习,在阿里飞猪数据集上优于单/多模态基线,提升POI标注性能。

摘要:

POI标注旨在为兴趣点(POI)附加一些信息丰富的标签,这有助于许多与POI相关的服务,包括搜索、推荐等。现有的大多数解决方案忽视了POI图像的重要性,很少融合POI的文本和视觉特征,导致标注性能欠佳。在本文中,我们提出了一种新颖的用于POI标注的多模态模型,即M3PT,它通过融合目标POI的文本和视觉特征,以及多模态表示之间的精确匹配,实现了增强的POI标注。具体来说,我们首先设计了一个领域自适应图像编码器(DIE),以获得与其黄金标签语义对齐的图像嵌入。然后,在M3PT的文本-图像融合模块(TIF)中,文本和视觉表示被充分融合到POI的内容嵌入中,以供后续匹配。此外,我们采用对比学习策略来进一步弥合不同模态表示之间的差距。为了评估标注模型的性能,我们从阿里飞猪的真实业务场景中构建了两个高质量的POI标注数据集。在这些数据集上,我们进行了广泛的实验,以证明我们的模型相对于单模态和多模态基线的优势,并验证M3PT中重要组件(包括DIE、TIF和对比学习策略)的有效性。

英文摘要:

POI tagging aims to annotate a point of interest (POI) with some informative tags, which facilitates many services related to POIs, including search, recommendation, and so on. Most of the existing solutions neglect the significance of POI images and seldom fuse the textual and visual features of POIs, resulting in suboptimal tagging performance. In this paper, we propose a novel Multi-Modal Model for POI Tagging, namely M3PT, which achieves enhanced POI tagging through fusing the target POI's textual and visual features, and the precise matching between the multi-modal representations. Specifically, we first devise a domain-adaptive image encoder (DIE) to obtain the image embeddings aligned to their gold tags' semantics. Then, in M3PT's text-image fusion module (TIF), the textual and visual representations are fully fused into the POIs' content embeddings for the subsequent matching. In addition, we adopt a contrastive learning strategy to further bridge the gap between the representations of different modalities. To evaluate the tagging models' performance, we have constructed two high-quality POI tagging datasets from the real-world business scenario of Ali Fliggy. Upon the datasets, we conducted the extensive experiments to demonstrate our model's advantage over the baselines of uni-modality and multi-modality, and verify the effectiveness of important components in M3PT, including DIE, TIF and the contrastive learning strategy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑