arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.13945cs.AIcs.CLcs.LG
收藏

CityBench:评估大型语言模型在城市任务中的能力

CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks

Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、北京理工大学、清华大学);School of Electronic and Information Engineering, Beijing Jiaotong University(电子信息工程学院、北京交通大学);Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院、清华大学)

Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, Yong Li

AI总结:

本文提出首个基于交互式模拟器的城市任务评估基准CityBench,整合多源城市数据与细粒度动态模拟,设计8项感知与决策任务,对30个模型测试发现其在常识语义任务表现优异,但在专业知识与数值计算任务上存在不足。

摘要:

随着大型语言模型(LLMs)不断进步并获得广泛应用,建立系统且可靠的LLMs和视觉语言模型(VLMs)评估方法已成为确保其在现实世界中有效性和可靠性的关键。目前已有一些关于LLMs在有限城市任务中可用性的早期探索,但仍缺乏系统且可扩展的评估基准。构建城市研究系统评估基准的挑战在于城市数据的多样性、应用场景的复杂性以及城市环境的高度动态性。本文设计了基于交互式模拟器的评估平台CityBench,作为首个评估LLMs在城市研究中多样化任务能力的系统基准。首先,我们构建CityData以整合多样化的城市数据,并构建CitySimu以模拟细粒度的城市动态。基于CityData和CitySimu,我们设计了感知-理解与决策两大类下的8项代表性城市任务作为CityBench。通过在全球13个城市对30个知名LLMs和VLMs进行广泛测试,我们发现先进的LLMs和VLMs在需要常识和语义理解能力的多样化城市任务中能取得具竞争力的表现,例如理解人类动态和对城市图像进行语义推理。同时,它们未能解决需要专业知识和高级数值能力的挑战性城市任务,例如地理空间预测和交通控制任务。

英文摘要:

As large language models (LLMs) continue to advance and gain widespread use, establishing systematic and reliable evaluation methodologies for LLMs and vision-language models (VLMs) has become essential to ensure their real-world effectiveness and reliability. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design \textit{CityBench}, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build \textit{CityData} to integrate the diverse urban data and \textit{CitySimu} to simulate fine-grained urban dynamics. Based on \textit{CityData} and \textit{CitySimu}, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the \textit{CityBench}. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level numerical abilities, e.g., geospatial prediction and traffic control task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑