arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2510.17685cs.CVcs.AI

通过双向关系推理与对齐实现多语言文本到图像行人检索

Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning

School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);School of Computer Science, Wuhan University(武汉大学计算机学院);Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新 generation 人工智能技术及交叉应用重点实验室(东南大学),教育部,中国)

Min Cao, Xinyu Zhou, Ding Jiang, Bo Du, Mang Ye, Min Zhang

AI总结:

该研究构建多语言TIPR基准,并提出Bi-IRRA框架,通过双向隐式关系推理与多维全局对齐实现跨语言、跨模态行人检索,取得最新最优性能。

摘要:

文本到图像行人检索(TIPR)旨在利用文本描述识别目标行人,但面临模态异质性挑战。此前工作尝试通过开发跨模态全局或局部对齐策略来解决该问题。然而,全局方法通常忽略细粒度跨模态差异,而局部方法需要先验信息来探索显式部位对齐。此外,当前方法以英语为中心,限制了其在多语言场景中的应用。为缓解这些问题,我们通过构建多语言TIPR基准开创了多语言TIPR任务:利用大语言模型生成初始翻译,并结合领域特定知识对其进行优化。相应地,我们提出Bi-IRRA,这是一个双向隐式关系推理与对齐框架,用于学习跨语言和跨模态的对齐。在Bi-IRRA中,双向隐式关系推理模块支持对掩码图像和文本进行双向预测,隐式增强跨语言、跨模态局部关系建模;同时集成多维全局对齐模块以弥合模态异质性。所提方法在所有多语言TIPR数据集上均取得最新最优结果。数据和代码见https://github.com/Flame-Chasers/Bi-IRRA。

英文摘要:

Text-to-image person retrieval (TIPR) aims to identify the target person using textual descriptions, facing challenge in modality heterogeneity. Prior works have attempted to address it by developing cross-modal global or local alignment strategies. However, global methods typically overlook fine-grained cross-modal differences, whereas local methods require prior information to explore explicit part alignments. Additionally, current methods are English-centric, restricting their application in multilingual contexts. To alleviate these issues, we pioneer a multilingual TIPR task by developing a multilingual TIPR benchmark, for which we leverage large language models for initial translations and refine them by integrating domain-specific knowledge. Correspondingly, we propose Bi-IRRA: a Bidirectional Implicit Relation Reasoning and Aligning framework to learn alignment across languages and modalities. Within Bi-IRRA, a bidirectional implicit relation reasoning module enables bidirectional prediction of masked image and text, implicitly enhancing the modeling of local relations across languages and modalities, a multi-dimensional global alignment module is integrated to bridge the modality heterogeneity. The proposed method achieves new state-of-the-art results on all multilingual TIPR datasets. Data and code are presented in https://github.com/Flame-Chasers/Bi-IRRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑