通过可微认知引导的多模态大语言模型进行生成式人类-物体交互检测
Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
AI总结：
本文提出GRASP-HO框架，通过可微认知引导的多模态大语言模型实现生成式人类-物体交互检测，解决封闭集分类与开放词汇生成之间的监督不匹配问题，实现判别感知与生成推理的统一。
摘要：
人类-物体交互（HOI）检测旨在定位人类-物体对及其相互作用。现有方法基于封闭世界假设，将任务视为在小预定义动词集上的分类问题，难以泛化到野外的长尾或模糊交互。尽管最近的多模态大语言模型（MLLMs）具备丰富的世界知识，用于开放词汇理解，但它们仍与现有HOI检测器脱节，因为微调它们计算上不可行。为了解决这些限制，我们提出\GRASP-HO}，一种新的生成推理和可引导感知框架，将HOI检测从封闭集分类任务转换为开放词汇生成问题。为了弥合视觉与认知之间的鸿沟，我们首先提取混合交互表示，然后设计一个轻量级可学习认知引导通道（CSC）模块，将细粒度的视觉证据注入冻结的MLLM中以实现有效的推理。为了解决基于分类的HOI数据集与开放词汇生成模型之间的监督不匹配问题，我们引入了一种混合指导策略，将语言建模损失和辅助分类损失耦合在一起，从而在不牺牲生成灵活性的情况下实现判别性定位。实验表明，该方法在封闭集性能上达到最先进的水平，并在零样本泛化方面表现强劲，实现了无缝连接判别感知与生成推理的统一范式，用于开放世界HOI检测。
英文摘要：
Human-object interaction (HOI) detection aims to localize human-object pairs and the interactions between them. Existing methods operate under a closed-world assumption, treating the task as a classification problem over a small, predefined verb set, which struggles to generalize to the long-tail of unseen or ambiguous interactions in the wild. While recent multi-modal large language models (MLLMs) possess the rich world knowledge required for open-vocabulary understanding, they remain decoupled from existing HOI detectors since fine-tuning them is computationally prohibitive. To address these constraints, we propose \GRASP-HO}, a novel Generative Reasoning And Steerable Perception framework that reformulates HOI detection from the closed-set classification task to the open-vocabulary generation problem. To bridge the vision and cognitive, we first extract hybrid interaction representations, then design a lightweight learnable cognitive steering conduit (CSC) module to inject the fine-grained visual evidence into a frozen MLLM for effective reasoning. To address the supervision mismatch between classification-based HOI datasets and open-vocabulary generative models, we introduce a hybrid guidance strategy that coupling the language modeling loss and auxiliary classification loss, enabling discriminative grounding without sacrificing generative flexibility. Experiments demonstrate state-of-the-art closed-set performance and strong zero-shot generalization, achieving a unified paradigm that seamlessly bridges discriminative perception and generative reasoning for open-world HOI detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
