arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.19913cs.CLcs.AIcs.LGcs.RO
收藏

MANGO:评估大型语言模型地图构建与导航能力的基准

MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models

University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Syracuse University(雪城大学);WeChat AI(微信人工智能);New Jersey Institute of Technology(新泽西理工学院)

Peng Ding, Jiading Fang, Peng Li, Kangrui Wang, Xiaochen Zhou, Mo Yu, Jing Li, Matthew R. Walter, Hongyuan Mei

AI总结:

本文提出 MANGO 基准,通过 53 个文本迷宫及其走法记录上的问答任务评估大型语言模型的地图构建与导航能力,发现 GPT-4 表现不佳,并证明该能力有助于下游文本游戏任务。

摘要:

大型语言模型(如 ChatGPT 和 GPT-4)最近在多种自然语言处理任务上取得了惊人的表现。本文提出了 MANGO,一个用于评估它们执行基于文本的地图构建与导航能力的基准。该基准包含 53 个取自一套文本游戏的迷宫:每个迷宫配有一条遍历所有地点但不覆盖所有可能路径的走法记录。任务形式为问答:对于每个迷宫,大型语言模型阅读走法记录并回答数百个地图构建与导航问题,例如“从 House 的西侧应如何前往 Attic?”以及“如果我们从 Cellar 向北再向东走,我们会到达哪里?”。尽管这些问题对人类来说很容易,但结果表明,即使是目前最先进的语言模型 GPT-4,在回答这些问题时也表现不佳。此外,我们的实验表明,强大的地图构建与导航能力将有助于大型语言模型执行相关的下游任务,例如玩文本游戏。我们的 MANGO 基准将促进未来关于提升语言模型地图构建与导航能力的方法研究。我们在 https://mango.ttic.edu 和 https://github.com/oaklight/mango/ 上托管排行榜、数据、代码和评估程序。

英文摘要:

Large language models such as ChatGPT and GPT-4 have recently achieved astonishing performance on a variety of natural language processing tasks. In this paper, we propose MANGO, a benchmark to evaluate their capabilities to perform text-based mapping and navigation. Our benchmark includes 53 mazes taken from a suite of textgames: each maze is paired with a walkthrough that visits every location but does not cover all possible paths. The task is question-answering: for each maze, a large language model reads the walkthrough and answers hundreds of mapping and navigation questions such as "How should you go to Attic from West of House?" and "Where are we if we go north and east from Cellar?". Although these questions are easy to humans, it turns out that even GPT-4, the best-to-date language model, performs poorly at answering them. Further, our experiments suggest that a strong mapping and navigation ability would benefit large language models in performing relevant downstream tasks, such as playing textgames. Our MANGO benchmark will facilitate future research on methods that improve the mapping and navigation capabilities of language models. We host our leaderboard, data, code, and evaluation program at https://mango.ttic.edu and https://github.com/oaklight/mango/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑