arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.13614cs.CVcs.AIcs.CLcs.IRcs.MM
收藏

面向像素级视觉实体链接的逆向区域到实体标注

Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking

Beijing Institute of Technology(北京理工大学);Tencent(腾讯)

Zhengfei Xu, Sijia Zhao, Yanchao Hao, Xiaolong Liu, Lili Li, Yuyang Yin, Bo Li, Xi Chen, Xin Xin

AI总结:

提出像素级视觉实体链接(PL-VEL)任务,构建含超500万标注的MaskOVEN-Wiki数据集,并提出视觉语义分词方法,显著提升模型准确率。

摘要:

视觉实体链接(VEL)是实现细粒度视觉理解的关键任务,它将图像内的物体(视觉提及)与知识库中的实体进行匹配。以往的VEL任务依赖文本输入,但为复杂场景编写查询语句可能具有挑战性。点击或边界框等视觉输入提供了一种更便捷的替代方案。因此,我们提出了一项新任务——像素级视觉实体链接(PL-VEL),该任务使用来自视觉输入的像素掩码来指代物体,为VEL补充了指代方法。为了促进该任务的研究,我们通过完全自动的逆向区域-实体标注框架构建了MaskOVEN-Wiki数据集。该数据集包含超过500万个将像素级区域与实体级标签对齐的标注,将推动视觉理解向细粒度发展。此外,由于像素掩码对应图像中的语义区域,我们通过视觉语义分词方法,利用区域交互注意力增强了先前的图块交互注意力。人工评估结果表明,该逆向标注框架达到了94.8%的标注成功率。实验结果显示,在该数据集上训练的模型相比零样本模型准确率提升了18个百分点。此外,语义分词方法相比已训练的基线模型实现了5个百分点的准确率提升。

英文摘要:

Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, but writing queries for complex scenes can be challenging. Visual inputs like clicks or bounding boxes offer a more convenient alternative. Therefore, we propose a new task, Pixel-Level Visual Entity Linking (PL-VEL), which uses pixel masks from visual inputs to refer to objects, supplementing reference methods for VEL. To facilitate research on this task, we have constructed the MaskOVEN-Wiki dataset through an entirely automatic reverse region-entity annotation framework. This dataset contains over 5 million annotations aligning pixel-level regions with entity-level labels, which will advance visual understanding towards fine-grained. Moreover, as pixel masks correspond to semantic regions in an image, we enhance previous patch-interacted attention with region-interacted attention by a visual semantic tokenization approach. Manual evaluation results indicate that the reverse annotation framework achieved a 94.8% annotation success rate. Experimental results show that models trained on this dataset improved accuracy by 18 points compared to zero-shot models. Additionally, the semantic tokenization method achieved a 5-point accuracy improvement over the trained baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑