MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models
University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Syracuse University(雪城大学);WeChat AI(微信人工智能);New Jersey Institute of Technology(新泽西理工学院)
Peng Ding, Jiading Fang, Peng Li, Kangrui Wang, Xiaochen Zhou, Mo Yu, Jing Li, Matthew R. Walter, Hongyuan Mei
Large language models such as ChatGPT and GPT-4 have recently achieved astonishing performance on a variety of natural language processing tasks. In this paper, we propose MANGO, a benchmark to evaluate their capabilities to perform text-based mapping and navigation. Our benchmark includes 53 mazes taken from a suite of textgames: each maze is paired with a walkthrough that visits every location but does not cover all possible paths. The task is question-answering: for each maze, a large language model reads the walkthrough and answers hundreds of mapping and navigation questions such as "How should you go to Attic from West of House?" and "Where are we if we go north and east from Cellar?". Although these questions are easy to humans, it turns out that even GPT-4, the best-to-date language model, performs poorly at answering them. Further, our experiments suggest that a strong mapping and navigation ability would benefit large language models in performing relevant downstream tasks, such as playing textgames. Our MANGO benchmark will facilitate future research on methods that improve the mapping and navigation capabilities of language models. We host our leaderboard, data, code, and evaluation program at https://mango.ttic.edu and https://github.com/oaklight/mango/.
Beyond Spatial Benchmarks: From Spatial Reasoning to Navigation
Xiamen University(厦门大学);Zhongguancun Academy(中关村学院);Beihang University(北京航空航天大学);Beijing Institute of Technology(北京理工大学);University of Chinese Academy of Sciences(中国科学院大学)
GUICourse: From General Vision Language Models to Versatile GUI Agents
Renmin University of China(中国人民大学);Tsinghua University(清华大学);Xiamen University(厦门大学);BUPT(北邮);ModelBest;Chinese Academy of Sciences(中国科学院);National University of Singapore(新加坡国立大学);Shanghai Qi Zhi Institute(上海启智研究院)
Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Ant Group(蚂蚁集团);CUHK MMLab(香港中文大学MMLab);Nanjing University(南京大学)
From Prompts to Protection: Large Language Model-Enabled In-Context Learning for Smart Public Safety UAV
Real-Time and Embedded Computing Systems Research Centre (CISTER)(实时嵌入式计算系统研究中心);Carnegie Mellon University(卡内基梅隆大学);Pontificia Universidad Católica de Chile(天主教大学);School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Instituto de Telecomunicações, Faculdade de Engenharia, Universidade do Porto(电信研究所,工程学院,葡萄牙里斯本大学);University of Houston(休斯顿大学)
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息多媒体国家重点实验室,计算机学院,北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute for Brain and Intelligence, Fudan University(脑与智能研究院,复旦大学);University of Science and Technology Beijing(北京科技大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心)
Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Pennsylvania(宾夕法尼亚大学);University of California San Diego(加州大学圣地亚哥分校);University of Michigan(密歇根大学);Amazon AGI(亚马逊人工通用智能)