arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.36384cs.CVcs.AIcs.CLq-bio.NC
收藏

SOMA:用于细粒度自我中心视角行为与身体运动理解的基准测试集

SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)

Matea Tashkovska, Naël Ouerghemi, Haozhe Qi, Alexander Mathis

AI总结:

本文提出SOMA基准测试集,评估MLLMs对自我中心视角细粒度身体运动的理解能力,发现其表现逊于人类,还引入训练数据集与多模态基线以推动相关研究

摘要:

感知人体运动对于理解和辅助人类的多模态系统至关重要。然而,当前对多模态大语言模型(MLLMs)的评估大多聚焦于物体、场景和粗粒度动作,却未充分评估其对身体本身的推理能力,从自我中心视角开展的评估尤其具有挑战性——该视角下身体仅部分可见,且运动需从被遮挡、时间分布的线索中推断。本文提出SOMA,一个用于从自我中心视角视频中理解细粒度身体运动的基准测试集,包含经人工验证的1632个问题,覆盖身体构型、运动动力学、时间推理三个能力维度。我们评估了37个领先的闭源、开源及专用自我中心视角MLLMs,并补充了广泛的人工评估。结果显示,人类仅通过视频完成该任务也存在难度(准确率68.5%),但明显优于最强的MLLM(Qwen3.6-27B),后者的零样本准确率为50.0%。错误分析表明,MLLMs常误判佩戴者自身身体、时间事件,且易出现偏向右手、低估重复次数等偏差。为缩小与人类表现的差距并推动进一步研究,我们引入了对齐视频、身体运动与语言监督的训练数据集,以及从这些模态联合学习的多模态基线。通过识别当前MLLMs的常见失败模式与缺失能力,我们希望SOMA能启发未来关于身体运动理解与多模态推理的研究。完整基准测试集、训练数据及代码将发布于:this https URL

英文摘要:

Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone (68.5% accuracy), yet clearly outperform the strongest MLLM (Qwen3.6-27B), which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. Full benchmark, training data, and code will be released at: https://cnai.epfl.ch/soma

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑