arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.07765cs.CLcs.AI
收藏

GameTraversalBenchmark:通过遍历2D游戏地图评估大型语言模型的规划能力

GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps

University of the Witwatersrand(威特沃特斯兰德大学);New York University(纽约大学)

Muhammad Umair Nasir, Steven James, Julian Togelius

AI总结:

本文提出GameTraversalBenchmark(GTB),通过2D网格游戏地图评估LLMs的规划能力,发现GPT-4-Turbo得分44.97%,推理模型o1得分67.84%,表明该基准仍具挑战性。

摘要:

大型语言模型(LLMs)近期在生成和理解自然语言方面展现了巨大成功。尽管它们在自然语言领域之外也显示出潜力,但这些LLMs能在多大程度上以及以何种方式进行规划,仍是一个悬而未决的问题。我们通过提出GameTraversalBenchmark(GTB)来调查其规划能力,这是一个由多样化基于2D网格的游戏地图组成的基准。如果LLM能以最少的步数和最少的生成错误遍历给定目标,则判定其成功。我们在GTB上评估了多个LLM,发现GPT-4-Turbo在GTB_Score(GTBS)上取得了44.97%的最高分,该分数是结合上述三个标准的综合得分。此外,我们初步测试了大型推理模型,即o1,其在GTBS上得分67.84%,表明该基准对当前模型仍具挑战性。代码、数据和文档可在https://github.com/umair-nasir14/Game-Traversal-Benchmark获取。

英文摘要:

Large language models (LLMs) have recently demonstrated great success in generating and understanding natural language. While they have also shown potential beyond the domain of natural language, it remains an open question as to what extent and in which way these LLMs can plan. We investigate their planning capabilities by proposing GameTraversalBenchmark (GTB), a benchmark consisting of diverse 2D grid-based game maps. An LLM succeeds if it can traverse through given objectives, with a minimum number of steps and a minimum number of generation errors. We evaluate a number of LLMs on GTB and found that GPT-4-Turbo achieved the highest score of 44.97% on GTB\_Score (GTBS), a composite score that combines the three above criteria. Furthermore, we preliminarily test large reasoning models, namely o1, which scores $67.84\%$ on GTBS, indicating that the benchmark remains challenging for current models. Code, data, and documentation are available at https://github.com/umair-nasir14/Game-Traversal-Benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑