arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.15862cs.CL
收藏

SportQA:评估大型语言模型体育理解能力的基准

SportQA: A Benchmark for Sports Understanding in Large Language Models

University of California, Irvine(加州大学欧文分校);Stanford University(斯坦福大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Meta Platforms, Inc.(元平台公司);Beijing Normal University(北京师范大学)

Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, Weining Shen

AI总结:

本文构建含逾7万道三级难度选择题的 SportQA 基准,结合少样本学习与 CoT 提示评估主流 LLM,发现其基础体育知识尚可但复杂场景推理落后于人类。

摘要:

深入理解体育这一富含策略性与动态内容的领域,对推进自然语言处理(NLP)至关重要。鉴于当前专业基准的缺失,这一点在评估和改进大型语言模型(LLM)方面尤为重要。为弥补这一空白,我们推出 SportQA,这是一个专门用于评估 LLM 体育理解能力的新型基准。SportQA 包含超过 70,000 道选择题,分为三个不同难度层级,分别针对体育知识的不同方面,从基础历史事实到复杂的、基于场景的推理任务。我们主要采用少样本学习范式,并辅以思维链(CoT)提示,对主流 LLM 进行了全面评估。结果表明,尽管 LLM 在基础体育知识方面表现尚可,但在更复杂的、基于场景的体育推理上存在困难,落后于人类专业水平。SportQA 的提出标志着 NLP 领域迈出重要一步,为评估和增强 LLM 的体育理解能力提供了工具。

英文摘要:

A deep understanding of sports, a field rich in strategic and dynamic content, is crucial for advancing Natural Language Processing (NLP). This holds particular significance in the context of evaluating and advancing Large Language Models (LLMs), given the existing gap in specialized benchmarks. To bridge this gap, we introduce SportQA, a novel benchmark specifically designed for evaluating LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenario-based reasoning tasks. We conducted a thorough evaluation of prevalent LLMs, mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting. Our results reveal that while LLMs exhibit competent performance in basic sports knowledge, they struggle with more complex, scenario-based sports reasoning, lagging behind human expertise. The introduction of SportQA marks a significant step forward in NLP, offering a tool for assessing and enhancing sports understanding in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑