SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models
Rice University(莱斯大学);University of California, Irvine(加州大学欧文分校);George Mason University(乔治梅森大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Stanford University(斯坦福大学)
Haotian Xia, Zhengbang Yang, Junbo Zou, Rhys Tracy, Yuqing Wang, Chi Lu, Christopher Lai, Yanjun He, Xun Shao, Zhuoqing Xie, Yuan-fang Wang, Weining Shen, Hanjie Chen
Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluate four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. We evaluate four LLMs using few-shot learning and chain-of-thought (CoT) prompting on SPORTU-text. GPT-4o achieves the highest accuracy of 71%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 7 proprietary and 6 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. Claude-3.5-Sonnet performs the best with only 52.6% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning.
SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports
Department of Computer Science, Rice University(Rice大学计算机科学系);Ken Kennedy Institute, Rice University(Rice大学肯尼迪研究所);Department of Statistics, University of California, Irvine(伊利诺伊大学欧文分校统计系);College of Sciences, Georgia Institute of Technology(佐治亚理工学院科学学院);Department of Applied Mathematics and Statistics, Johns Hopkins University(约翰霍普金斯大学应用数学与统计学系);Department of Computer Science, University of California, Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology(北京智能信息科技重点实验室,计算机科学与技术学院,北京理工大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);State Key Laboratory of General Artificial Intelligence, Peking University(通用人工智能国家重点实验室,北京大学);Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University(广东机器感知与智能计算实验室,深圳MSU-BIT大学);Department of Automation, Tsinghua University(自动化系,清华大学)
FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoning
School of Automation, Northwestern Polytechnical University(自动化学院,西北工业大学);The University of Hong Kong(香港大学);School of Software, Northwestern Polytechnical University(软件学院,西北工业大学);Unmanned System Research Institute, Northwestern Polytechnical University(无人系统研究院,西北工业大学)
Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
Indian Institute of Technology Patna(印度理工学院帕纳布分校);Sardar Patel Institute of Technology(萨达尔·帕特尔技术学院);Universitas Gadjah Mada(加查马大学);King Mongkut’s Institute of Technology Ladkrabang(拉差班国王技术学院);Shenzhen Technology University(深圳技术大学);Université de Toulouse(图卢兹大学)
Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning
Beijing University of Technology(北京理工大学);Shanghai Jiao Tong University(上海交通大学);Chinese Academy of Sciences(中国科学院);The Hong Kong Polytechnic University(香港理工大学)
$\left|\,\circlearrowright\,\boxed{\text{BUS}}\,\right|$: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles
Tredence Inc.(特伦德公司);Indian Institute of Technology Kharagpur(印度理工学院克拉格浦尔分校);Inria Paris-Rocquencourt(巴黎-罗克琴库特研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒姆研究实验室)
DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning
Georgia Institute of Technology(佐治亚理工学院);Rice University(Rice大学);Johns Hopkins University(约翰霍普金斯大学);University of California, Irvine(加州大学 Irvine分校);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Towards Temporal Compositional Reasoning in Long-Form Sports Videos
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS)
FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding
Lappeenranta-Lahti University of Technology LUT(拉佩兰塔-拉赫蒂技术大学);Tianjin University(天津大学);AMD Silo AI;Great Bay University(大湾大学);Rensselaer Polytechnic Institute(雷士大学);Brno University of Technology(布拉格技术大学)
SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding
SimulaMet and OsloMet, Norway(SimulaMet和OsloMet,挪威);Forzasys, Norway(Forzasys,挪威);SimulaMet, Norway(SimulaMet,挪威);SimulaMet, OsloMet, and Forzasys, Norway(SimulaMet、OsloMet和Forzasys,挪威);University of Central Florida, USA(佛罗里达中央大学)
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University(复旦大学);NJU-LINK Team, Nanjing University(南京大学NJU-LINK团队);University of Science and Technology of China(中国科学技术大学);Kuaishou Technology(快手科技)
A Unified Taxonomy and Multimodal Dataset for Events in Invasion Games
German Sport University Cologne(科隆德国体育大学);Leibniz University Hannover(莱布尼茨汉诺威大学);TIB – Leibniz Information Centre for Science and Technology(TIB – 莱布尼茨科学技术信息中心)
PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);University of California San Diego(加州大学圣地亚哥分校);Australian National University(澳大利亚国立大学)
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
School of Intelligence Science and Technology, Peking University(北京理工大学智能科学与技术学院);Wuhan University(武汉大学);Shanghai Jiao Tong University(上海交通大学);Nanyang Technological University(南洋理工大学);CASIA(中国科学院自动化研究所);University of Tokyo(东京大学);University of Liverpool(利物浦大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);UC Merced(加州大学默塞德分校)
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
FAIR, Meta(Meta FAIR实验室);University of Texas at Austin(德克萨斯大学奥斯汀分校);Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加利福尼亚大学伯克利分校);Project Aria, Meta(Meta Project Aria);University of Bristol(布里斯托尔大学);IIIT, Hyderabad(印度信息技术研究所海得拉巴分校);University of Illinois, Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Minnesota(明尼苏达大学);National University of Singapore(新加坡国立大学);Universidad de los Andes(安第斯大学);Simon Fraser University(西蒙菲莎大学);University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校)