语义模态补偿用于非配对设置下的无监督可见光-红外行人重识别

中文摘要 AI 辅助

无监督可见光-红外行人重识别（USL-VI-ReID）旨在无需身份标注的情况下学习可跨模态比较的行人表示。然而，在非配对设置下，模态间的身份对应关系往往不完整，导致许多身份在另一模态中缺乏可观测的对应样本。现有的非配对方法通过生成或映射另一模态的特征来弥合这一差距，主要利用视觉特征的统计信息，而未明确区分对身份具有判别性的内容与模态特有的风格。因此，生成的特征可能扭曲身份线索或继承源模态的偏差，削弱跨模态监督的可靠性。我们将跨模态非配对学习表述为一个语义补偿问题，并提出语义模态补偿（SMC）框架，该框架基于提示组合，在共享的视觉语义空间中将身份语义与模态风格解耦。SMC首先通过增强的双重对比学习构建判别性ReID空间，为每个模态生成伪标签、聚类原型和记忆库。然后，它在CLIP语义空间中学习可见光和红外模态提示，并将从伪标签获得的聚类映射为身份语义标记。对于在另一模态中缺乏可靠匹配的每个聚类，SMC将其身份标记与目标模态的提示相结合，以在缺失模态中合成语义对应样本。随后，合成的对应样本被投影回ReID空间，并通过置信度门控注入补偿记忆。在配对和非配对设置下的大量实验表明，SMC始终优于最先进的方法，尤其在身份不匹配严重时提升显著。

英文摘要

Unsupervised visible-infrared person re-identification (USL-VI-ReID) learns person representations that can be compared across modalities without identity annotations. In the unpaired setting, however, identity correspondences between modalities are often incomplete, leaving many identities without an observed counterpart in the other modality. Existing unpaired methods bridge this gap by generating or mapping features for the other modality, mainly by exploiting the statistics of visual features without explicitly separating content that is discriminative for identity from style that is specific to modality. Consequently, the generated features may distort identity cues or inherit bias from the source modality, undermining the reliability of supervision across modalities. We formulate unpaired learning across modalities as a semantic compensation problem and propose Semantic Modality Compensation (SMC), a framework based on prompt composition that decouples identity semantics from modality style within a shared visual semantic space. SMC first constructs a discriminative ReID space through augmented dual contrastive learning, yielding pseudo labels, cluster prototypes, and memory banks for each modality. It then learns visible and infrared modality prompts in the CLIP semantic space and maps clusters obtained from pseudo labels to identity semantic tokens. For each cluster lacking a reliable match in the other modality, SMC combines its identity token with the prompt for the target modality to synthesize a semantic counterpart in the missing modality. The synthesized counterpart is then projected back into the ReID space and injected into a compensation memory through confidence gating. Extensive experiments under both paired and unpaired settings demonstrate that SMC consistently outperforms state-of-the-art methods, with particularly large gains when identity mismatch is severe.

发表机构

• Wuhan University（武汉大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Preprint