利用文本组合推理实现鲁棒的变更描述
Leveraging Textual Compositional Reasoning for Robust Change Captioning
AI总结：
CORTEX通过整合文本线索和组合推理，提升图像变化描述的鲁棒性与准确性。
摘要：
变更描述旨在描述一对图像之间的变化。然而，现有方法仅依赖视觉特征，这往往无法捕捉微妙但有意义的变化，因为它们缺乏表示显式结构信息如对象关系和组合语义的能力。为了解决这个问题，我们提出了CORTEX（COmpositional Reasoning-aware TEXt-guided），一种新的框架，整合互补的文本线索以增强变化理解。除了捕捉像素级差异的线索外，CORTEX还利用由视觉语言模型（VLMs）提供的场景级文本知识，提取更丰富的图像文本信号，揭示底层的组合推理。CORTEX包含三个关键模块：（i）图像级变化检测器，用于识别配对图像之间的低级视觉差异；（ii）基于推理的文本提取（RTE）模块，利用VLMs生成隐含在视觉特征中的组合推理描述；（iii）图像-文本双对齐（ITDA）模块，对齐视觉和文本特征以实现细粒度的关系推理。这使CORTEX能够对视觉和文本特征进行推理，并捕捉那些在纯视觉特征中显得模糊的变化。
英文摘要：
Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to represent explicitly structured information such as object relationships and compositional semantics. To alleviate this, we present CORTEX (COmpositional Reasoning-aware TEXt-guided), a novel framework that integrates complementary textual cues to enhance change understanding. In addition to capturing cues from pixel-level differences, CORTEX utilizes scene-level textual knowledge provided by Vision Language Models (VLMs) to extract richer image text signals that reveal underlying compositional reasoning. CORTEX consists of three key modules: (i) an Image-level Change Detector that identifies low-level visual differences between paired images, (ii) a Reasoning-aware Text Extraction (RTE) module that use VLMs to generate compositional reasoning descriptions implicit in visual features, and (iii) an Image-Text Dual Alignment (ITDA) module that aligns visual and textual features for fine-grained relational reasoning. This enables CORTEX to reason over visual and textual features and capture changes that are otherwise ambiguous in visual features alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
