RefereeBench: 视频多模态大语言模型是否准备好成为多项目裁判
RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees
AI总结：
本文提出RefereeBench，首个评估多模态大语言模型作为自动体育裁判的基准，通过11项运动925个视频和6475对问答，评估犯规存在、分类、推理、实体感知和时间定位能力，发现当前模型在规则应用和时间定位上表现不足。
摘要：
尽管多模态大语言模型（MLLMs）在通用视频理解方面表现优异，但其支持专门化、基于规则的决策能力仍缺乏充分探索。本文介绍RefereeBench，首个大规模基准，涵盖11项运动925个精心挑选的视频和6,475对问答，评估五种核心裁判能力：犯规存在、犯规与罚球分类、犯规与罚球推理、实体感知和时间定位。该基准完全由人类标注，确保高质量标注基于真实裁判逻辑和多模态证据。对最先进的MLLMs的广泛评估显示，即使最强模型如Doubao-Seed-1.8和Gemini-3-Pro也仅达到约60%的准确率，而最强开源模型Qwen3-VL仅达到47%。这些结果表明，当前模型离成为可靠的体育裁判仍有差距。进一步分析显示，尽管模型能经常识别事件和相关实体，但在规则应用和时间定位上表现不足，并频繁在正常片段上误判犯规。本基准强调未来MLLMs需要更好地整合领域知识和多模态理解，推动可信的人工智能辅助裁判和更广泛的多模态决策。
英文摘要：
While Multimodal Large Language Models (MLLMs) excel at generic video understanding, their ability to support specialized, rule-grounded decision-making remains insufficiently explored. In this paper, we introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees. Spanning 11 sports with 925 curated videos and 6,475 QA pairs, RefereeBench evaluates five core officiating abilities: foul existence, foul and penalty classification, foul and penalty reasoning, entity perception, and temporal grounding. The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence. Extensive evaluations of state-of-the-art MLLMs show that even the strongest models, such as Doubao-Seed-1.8 and Gemini-3-Pro, achieve only around 60% accuracy, while the strongest open-source model, Qwen3-VL, reaches only 47%. These results indicate that current models remain far from being reliable sports referees. Further analysis shows that while models can often identify incidents and involved entities, they struggle with rule application and temporal grounding, and frequently over-call fouls on normal clips. Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding, advancing trustworthy AI-assisted officiating and broader multimodal decision-making.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
