探索大型基础模型在开放词汇 HOI 检测中的潜力
Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection
AI总结：
提出 CMD-SE 框架，通过条件式多层级解码处理不同距离的人-物对，并利用 LLM 生成人体部位细粒度语义，在 SWIG-HOI 与 HICO-DET 上达到开放词汇 HOI 检测 SOTA。
摘要：
开放词汇人-物交互（HOI）检测关注在自然语言引导下检测新 HOI 的问题，对于理解以人为中心的场景至关重要。然而，以往的零样本 HOI 检测器通常采用相同层级的特征图来建模距离不同的 HOI，导致在包含距离范围很广的人-物对的场景中性能欠佳。此外，这些检测器主要依赖类别名称，忽视了语言能够提供的丰富上下文信息，而这些信息对于捕捉开放词汇概念必不可少；这类概念通常十分罕见，仅凭类别名称无法得到充分表征。本文中，我们提出一种新颖的端到端开放词汇 HOI 检测框架，具备条件式多层级解码与细粒度语义增强（CMD-SE），以发挥视觉-语言模型（VLM）的潜力。具体而言，我们通过在二分匹配过程中引入软约束，提出使用不同层级的特征图来建模不同距离的人-物对。此外，通过利用 GPT 模型等大型语言模型（LLM），我们借助其广泛的世界知识，为各种交互生成人体部位状态的描述。随后，我们整合具有泛化性的细粒度人体部位语义，以改进交互识别。在 SWIG-HOI 和 HICO-DET 两个数据集上的实验结果表明，我们提出的方法在开放词汇 HOI 检测中取得了最先进结果。代码和模型可在 https://github.com/ltttpku/CMD-SE-release 获取。
英文摘要：
Open-vocabulary human-object interaction (HOI) detection, which is concerned with the problem of detecting novel HOIs guided by natural language, is crucial for understanding human-centric scenes. However, prior zero-shot HOI detectors often employ the same levels of feature maps to model HOIs with varying distances, leading to suboptimal performance in scenes containing human-object pairs with a wide range of distances. In addition, these detectors primarily rely on category names and overlook the rich contextual information that language can provide, which is essential for capturing open vocabulary concepts that are typically rare and not well-represented by category names alone. In this paper, we introduce a novel end-to-end open vocabulary HOI detection framework with conditional multi-level decoding and fine-grained semantic enhancement (CMD-SE), harnessing the potential of Visual-Language Models (VLMs). Specifically, we propose to model human-object pairs with different distances with different levels of feature maps by incorporating a soft constraint during the bipartite matching process. Furthermore, by leveraging large language models (LLMs) such as GPT models, we exploit their extensive world knowledge to generate descriptions of human body part states for various interactions. Then we integrate the generalizable and fine-grained semantics of human body parts to improve interaction recognition. Experimental results on two datasets, SWIG-HOI and HICO-DET, demonstrate that our proposed method achieves state-of-the-art results in open vocabulary HOI detection. The code and models are available at https://github.com/ltttpku/CMD-SE-release.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
