大型视觉-语言模型双边协作用于开放词汇人-物交互检测
Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
AI总结：
提出双边协作框架BC-HOI，通过注意力偏置引导和LLM监督引导，利用大型视觉-语言模型生成细粒度特征，提升开放词汇人-物交互检测性能。
摘要：
开放词汇人-物交互（HOI）检测是一项具有挑战性的任务，旨在检测图像中所有感兴趣的<人类，动词，物体>三元组，即使这些三元组未在训练集中预先定义。现有方法通常依赖大型视觉-语言模型（VLM）生成的输出特征来增强交互表示的泛化能力。然而，VLM产生的视觉特征是整体且粗粒度的，这与检测任务的性质相矛盾。为解决此问题，我们提出了一种新颖的双边协作框架用于开放词汇HOI检测（BC-HOI）。该框架包含一个注意力偏置引导（ABG）组件，该组件根据HOI检测器提供的注意力偏置，引导VLM生成细粒度的实例级交互特征。它还包含一个基于大型语言模型（LLM）的监督引导（LSG）组件，该组件通过VLM的LLM部分为HOI检测器提供细粒度的令牌级监督。LSG增强了ABG生成高质量注意力偏置的能力。我们在两个流行的基准测试上进行了广泛实验：HICO-DET和V-COCO，在开放词汇和封闭设置下均持续实现了优越的性能。代码将在Github上发布。
英文摘要：
Open vocabulary Human-Object Interaction (HOI) detection is a challenging task that detects all <human, verb, object> triplets of interest in an image, even those that are not pre-defined in the training set. Existing approaches typically rely on output features generated by large Vision-Language Models (VLMs) to enhance the generalization ability of interaction representations. However, the visual features produced by VLMs are holistic and coarse-grained, which contradicts the nature of detection tasks. To address this issue, we propose a novel Bilateral Collaboration framework for open vocabulary HOI detection (BC-HOI). This framework includes an Attention Bias Guidance (ABG) component, which guides the VLM to produce fine-grained instance-level interaction features according to the attention bias provided by the HOI detector. It also includes a Large Language Model (LLM)-based Supervision Guidance (LSG) component, which provides fine-grained token-level supervision for the HOI detector by the LLM component of the VLM. LSG enhances the ability of ABG to generate high-quality attention bias. We conduct extensive experiments on two popular benchmarks: HICO-DET and V-COCO, consistently achieving superior performance in the open vocabulary and closed settings. The code will be released in Github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
