arXiv 2604.015862026-04-02cs.CVcs.AI

SHOE：语义HOI开放词汇评估度量

SHOE: Semantic HOI Open-Vocabulary Evaluation Metric

University of Mississippi（密西西比大学）；National University of Singapore（新加坡国立大学）；Independent Researcher（独立研究员）；ASUS Intelligent Cloud Services (AICS)（华硕智能云服务（AICS））

Maja Noack, Qinqian Lei, Taipeng Tian, Bihan Dong, Robby T. Tan, Yixin Chen, John Young, Saijun Zhang, Bo Wang

备注：: Accepted to GRAIL-V Workshop at CVPR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SHOE评估框架，通过结合预测与真实HOI标签的语义相似性，改进开放词汇HOI检测的评估方法，实验表明其与人类判断一致率达85.73%。

摘要：

开放词汇人类-物体交互（HOI）检测是构建可扩展系统的关键步骤，这些系统能泛化到现实场景中的未见交互并支持基于人类-物体关系的多模态系统。然而，标准评估度量如平均平均精度（mAP）将HOI类别视为离散类别标签，无法认可语义有效但词汇不同的预测（例如“靠在沙发上”与“坐在沙发上”），限制了其在评估超出预定义HOI标签集的开放词汇预测中的应用。我们引入SHOE（语义HOI开放词汇评估），一种新的评估框架，结合预测与真实HOI标签之间的语义相似性。SHOE将每个HOI预测分解为动词和物体组成部分，利用多个大型语言模型（LLMs）的平均值估计其语义相似性，并将它们组合成相似性分数以评估超出精确字符串匹配的对齐情况。这使SHOE能够灵活且可扩展地评估现有HOI检测方法和开放生成模型，使用标准基准如HICO-DET。实验结果表明，SHOE得分更接近人类判断，包括基于LLM和嵌入基线的现有度量，与平均人类评分的同意率达85.73%。我们的工作强调了语义基础HOI评估的必要性，这种评估更好地反映了人类对交互的理解。我们将发布我们的评估度量以促进未来研究。

英文摘要：

Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human-object relationships. However, standard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., "lean on couch" vs. "sit on couch"), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the average human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。