MoveBench:用于评估大型语言模型在体育与健康指导领域表现的基于 rubric 的基准
MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching
AI总结:
MoveBench 是首个体育健康指导领域的 LLM 评估基准,含多任务形式与 rubric,经 15 种 LLM 评估发现其在安全等方面存差距,为相关 AI 指导系统提供标准与框架。
摘要:
尽管大型语言模型(LLMs)发展迅速,但由于缺乏领域特定的评估基准,其在体育与健康领域的部署仍存在高风险。现有的通用基准(如 MMLU 和 HELM)主要评估广泛的知识和推理能力,却无法捕捉运动安全、生理适宜性、个性化以及可操作的指导质量等关键领域需求。在健身与健康应用中,错误或不安全的建议可能导致身体伤害,这一差距尤其令人担忧。为解决这一局限,我们推出 MoveBench,这是首个专门设计用于评估 LLMs 在体育与健康指导领域表现的综合基准框架。该基准整合了多种任务形式,包括基于现实用户场景的多项选择题(MCQs)和开放式问答(OpenQA),并配有强调安全合规性、科学有效性和训练逻辑的结构化 rubric。通过对 15 种不同 LLMs 在 809 个问题及 5012 个 rubric 级标准上的大规模评估,我们的结果显示出显著的性能差异,尤其是在安全意识、结构化课程设计和个性化能力方面,凸显了“知与行”之间的关键差距。我们的工作为体育与健康领域的 LLM 应用建立了基础评估标准,并为推进安全、可靠且情境感知的 AI 驱动指导系统提供了可复现的框架。开放资源可在此 https URL 获取。
英文摘要:
Despite rapid advances in large language models (LLMs), their deployment in the sports and health domain remains high-risk due to the absence of domain-specific evaluation benchmarks. Existing general-purpose benchmarks (e.g., MMLU and HELM) primarily assess broad knowledge and reasoning capabilities, but fail to capture critical domain requirements such as exercise safety, physiological appropriateness, personalization, and actionable coaching quality. This gap is particularly concerning in fitness and health applications, where incorrect or unsafe recommendations can lead to physical harm. To address this limitation, we introduce MoveBench, the first comprehensive benchmarking framework specifically designed for evaluating LLMs in sports and health coaching. The proposed benchmark integrates heterogeneous task formats, including multiple-choice questions (MCQs) and open-ended question answering (OpenQA), grounded in real-world user scenarios and paired with structured rubrics emphasizing safety compliance, scientific validity, and training logic. Through large-scale evaluation of 15 diverse LLMs on 809 questions with 5,012 rubric-level criteria, our results reveal substantial performance variability, particularly in safety awareness, structured program design, and personalization capability, highlighting a critical gap between knowing and doing. Our work establishes a foundational evaluation standard for sports and health LLM applications and provides a reproducible framework for advancing safe, reliable, and context-aware AI-driven coaching systems. Open resources are available at https://github.com/SatoSakula/MoveBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。