arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.08474cs.CVcs.CL
收藏

SPORTU:面向多模态大语言模型的综合体育理解基准

SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models

Rice University(莱斯大学);University of California, Irvine(加州大学欧文分校);George Mason University(乔治梅森大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Stanford University(斯坦福大学)

Haotian Xia, Zhengbang Yang, Junbo Zou, Rhys Tracy, Yuqing Wang, Chi Lu, Christopher Lai, Yanjun He, Xun Shao, Zhuoqing Xie, Yuan-fang Wang, Weining Shen, Hanjie Chen

AI总结:

提出 SPORTU 多模态体育理解基准,通过文本 QA 与视频 QA 系统评估 LLM/MLLM 的多层次体育规则、策略和事件推理能力。

摘要:

多模态大语言模型(MLLM)正通过整合文本与视觉信息,提升对复杂体育场景进行推理的能力。为全面评估其能力,我们提出 SPORTU,这是一个旨在跨多层次体育推理任务评估 MLLM 的基准。SPORTU 包含两个关键组成部分:SPORTU-text 拥有 900 道选择题,并配有用于规则理解和策略理解的人工标注解释;该部分侧重于测试模型仅通过问答(QA)进行体育推理的能力,不需要视觉输入。SPORTU-video 由覆盖 7 项不同运动的 1,701 个慢动作视频片段以及 12,048 个 QA 对组成,旨在评估从简单的运动识别到犯规检测和规则应用等复杂任务的多层次推理。在 SPORTU-text 部分,我们评估了四款主流 LLM,主要采用少样本学习范式,并辅以思维链(CoT)提示。GPT-4o 达到最高的 71% 准确率,但仍低于人类水平,凸显出其在规则理解和推理方面仍有提升空间。SPORTU-video 部分的评估包括 7 款专有 MLLM 和 6 款开源 MLLM。实验表明,模型在需要深度推理和基于规则理解的困难任务上表现不足。Claude-3.5-Sonnet 表现最佳,但在困难任务上的准确率仅为 52.6%,显示出巨大的提升空间。我们希望 SPORTU 能成为评估模型在体育理解与推理方面能力的关键一步。

英文摘要:

Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluate four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. We evaluate four LLMs using few-shot learning and chain-of-thought (CoT) prompting on SPORTU-text. GPT-4o achieves the highest accuracy of 71%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 7 proprietary and 6 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. Claude-3.5-Sonnet performs the best with only 52.6% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑