NextMotionQA: 使用视觉语言模型基准测试和评判人体运动理解
NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
AI总结：
提出NextMotionQA基准，通过三项互补任务和多粒度难度分层，系统评估视觉语言模型在人体运动理解中的能力，并揭示其在细粒度评判中的局限性。
摘要：
人体运动理解的可靠评估对于推进具身人工智能、机器人和动画至关重要。然而，现有基准存在语义粒度粗糙、难度无区分、标注质量有限以及答案模糊等问题，无法诊断当前模型的失败之处。为弥补这一差距，我们引入NextMotionQA，这是一个全面的基准，利用视觉语言模型（VLM）进行半自动化、专家验证的数据集构建。NextMotionQA包含三项互补任务：多项选择题问答、视频字幕生成和细粒度错误纠正。每项任务沿三个核心语义轴系统组织，并分为三个任务复杂度级别。我们对十二个代表性VLM的广泛评估揭示了在传统单任务评估中不可见的关键能力差距和弱点。在互补方向上，近期工作开始使用VLM作为文本到运动评估的评判者；我们探究它们在更困难任务下是否表现出同样的退化。我们发现，VLM在粗粒度标准上与专家评分高度一致（Cohen's κ=0.70），但在细粒度、部件级评判上表现不佳（κ=0.10），验证了该范式在其强项领域的有效性，同时明确了其局限性。
英文摘要：
Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffer from coarse semantic granularity, undifferentiated difficulty, limited annotation quality, and pervasive answer ambiguity, leaving them unable to diagnose where current models fail. To bridge this gap, we introduce NextMotionQA, a comprehensive benchmark that leverages vision-language models (VLMs) for semi-automated, expert-verified dataset. NextMotionQA features three complementary tasks: multiple-choice question answering, video captioning, and fine-grained error correction. Each task is systematically structured across three core semantic axes and stratified into three task complexity levels. Our extensive evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations. In a complementary direction, recent work has begun using VLMs as judges for text-to-motion evaluation; we ask whether they show the same degradation under harder tasks. We find that VLMs align strongly with expert ratings on coarse criteria (Cohen's κ=0.70) but break down on fine-grained, part-level judgment (κ=0.10), validating the paradigm in its strong regime while clarifying its limits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
