探索基于视觉-语言模型的高效人-物交互检测中的交互语义对齐
Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model
AI总结：
提出ISA-HOI检测器，利用CLIP对齐交互语义，结合全局上下文与局部特征，改进图像交互特征和动词语义，在HICO-DET和V-COCO上以更少训练轮次取得竞争性能，并超越零样本SOTA。
摘要：
人-物交互（HOI）检测旨在定位人与物体对并理解其交互。近期，基于两阶段的Transformer方法展现了有竞争力的性能。然而，这些方法常聚焦于物体外观特征，忽略了全局上下文信息。此外，能有效对齐视觉和文本嵌入的视觉-语言模型CLIP在零样本HOI检测中展现出巨大潜力。基于上述事实，我们提出了一种名为ISA-HOI的新型HOI检测器，它广泛利用CLIP的知识，对齐视觉与文本特征之间的交互语义。我们首先提取图像的全局上下文和物体的局部特征，以改进图像中的交互特征（IF）。另一方面，我们提出一个动词语义改进（VSI）模块，通过跨模态融合增强动词标签的文本特征。最终，我们的方法在HICO-DET和V-COCO基准上以更少的训练轮次取得了有竞争力的结果，并在零样本设置下超越了现有最先进方法。
英文摘要：
Human-Object Interaction (HOI) detection aims to localize human-object pairs and comprehend their interactions. Recently, two-stage transformer-based methods have demonstrated competitive performance. However, these methods frequently focus on object appearance features and ignore global contextual information. Besides, vision-language model CLIP which effectively aligns visual and text embeddings has shown great potential in zero-shot HOI detection. Based on the former facts, We introduce a novel HOI detector named ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. We first extract global context of image and local features of object to Improve interaction Features in images (IF). On the other hand, we propose a Verb Semantic Improvement (VSI) module to enhance textual features of verb labels via cross-modal fusion. Ultimately, our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
