FineBench: 细粒度人类活动理解的视觉-语言模型基准测试与增强
FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
AI总结：
针对视觉-语言模型在细粒度人类活动理解上的不足，提出包含密集标注的长视频问答基准FineBench和增强框架FineAgent。
摘要：
视觉-语言模型（VLM）在通用视频理解方面表现出色，但在需要细致理解人类动作和交互的真实世界应用中，它们常常难以进行细粒度理解。虽然最近一些以人为中心的基准测试评估了模型行为的公平性/伦理、情感感知等维度，但它们没有结合长视频、密集的问答覆盖以及大规模的帧级空间/时间定位。为弥补这一差距，我们引入了FineBench，一个专门设计用于评估细粒度理解的以人为中心的视频问答（VQA）基准。FineBench包含199,420个多项选择问答对，密集标注在64个长视频（每个15分钟）上，重点关注详细的人物运动、人物交互和物体操作，包括组合动作。我们的广泛评估显示，虽然像GPT-5这样的专有模型取得了不错的性能，但当前的开源VLM明显表现不佳，特别是在多人场景的空间推理以及区分人类运动和交互的细微差异方面。为了解决这些已识别的弱点，我们提出了FineAgent，一个模块化框架，通过利用定位器和描述器来增强VLM。实验表明，FineAgent在FineBench上持续提高了各种开源VLM的性能。FineBench为未来细粒度以人为中心的视频理解研究提供了严格的测试平台，而FineAgent则为增强当前VLM中的此类推理提供了一种实用方法。项目页面和代码：https://joslefaure.github.io/assets/html/finebench.html。
英文摘要：
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in general video understanding, yet they often struggle with the fine-grained comprehension crucial for real-world applications requiring nuanced interpretation of human actions and interactions. While some recent human-centric benchmarks evaluate aspects of model behaviour such as fairness/ethics, emotion perception, and broader human-centric metrics, they do not combine long-form videos, very dense QA coverage, and frame-level spatial/temporal grounding at scale. To bridge this gap, we introduce FineBench, a human-centric video question answering (VQA) benchmark specifically designed to assess fine-grained understanding. FineBench comprises 199,420 multiple-choice QA pairs densely annotated across 64 long-form videos (15 minutes each), focusing on detailed person movement, person interaction, and object manipulation, including compositional actions. Our extensive evaluation reveals that while proprietary models like GPT-5 achieve respectable performance, current open-source VLMs significantly underperform, struggling particularly with spatial reasoning in multi-person scenes and distinguishing subtle differences in human movements and interactions. To address these identified weaknesses, we propose FineAgent, a modular framework that enhances VLMs by leveraging a Localizer and a Descriptor. Experiments show that FineAgent consistently improves the performance of various open VLMs on FineBench. FineBench provides a rigorous testbed for future research into fine-grained human-centric video understanding, while FineAgent offers a practical approach to enhance such reasoning in current VLMs. Project page and code at https://joslefaure.github.io/assets/html/finebench.html.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
