CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、北京理工大学、清华大学);School of Electronic and Information Engineering, Beijing Jiaotong University(电子信息工程学院、北京交通大学);Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院、清华大学)
Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, Yong Li
As large language models (LLMs) continue to advance and gain widespread use, establishing systematic and reliable evaluation methodologies for LLMs and vision-language models (VLMs) has become essential to ensure their real-world effectiveness and reliability. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design \textit{CityBench}, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build \textit{CityData} to integrate the diverse urban data and \textit{CitySimu} to simulate fine-grained urban dynamics. Based on \textit{CityData} and \textit{CitySimu}, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the \textit{CityBench}. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level numerical abilities, e.g., geospatial prediction and traffic control task.
UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models
The Thørväld Group(Thørväld集团);Inria Paris-Rocquencourt(Inria巴黎-罗克桑court分部);Rajiv Gandhi University Doimukh(拉贾格恩迪大学多伊穆克);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒默研究实验室);The Kumquat Consortium(Kumquat联盟)
CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)信息中心);Department of Electronic Engineering, BNRist, Tsinghua University(清华大学电子工程系);School of Electronic and Information Engineering, Beijing Jiaotong University(北京交通大学电子与信息工程学院)
TMBuD: A dataset for urban scene building detection
Politehnica University of Timişoara(蒂米什瓦拉理工大学)
城市场景中建筑物的识别和三维重建已成为图像处理领域的一个有趣而实际的课题。对于本研究主题,计算机视觉和增强现实领域交叉,以更好地理解各种主题的城市场景。在本文中,我们的目的是介绍一个数据集的解决方案,TMBuD,这是更好地适合图像处理的人造结构的城市场景场景。建议的数据集将允许适当的评价突出的边缘和语义分割的图像集中在街景角度的建筑物。形成我们数据集的图像提供了城市场景中建筑物的各种街景视角,这允许评估复杂的算法。该数据集包含来自罗马尼亚蒂米什瓦拉的160张建筑物图像,每张图像的分辨率为768 x 1024像素。
CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、BNRist、清华大学);School of Electronic and Information Engineering, Beijing Jiaotong University(电子信息工程学院、北京交通大学);Department of Electronic Engineering, Tsinghua University(电子工程系、清华大学);Department of Urban Planning, Tsinghua University(城市规划系、清华大学)
Towards Inclusive Mobility Modeling: Characterizing and Evaluating Elderly Trajectory Patterns in Urban Systems
School of Computing and Artificial Intelligence, Shanghai University of Finance and Economics(上海财经大学计算机与人工智能学院);MOE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics(上海财经大学计算与经济学交叉研究教育部重点实验室);Center for Data Science, New York University(纽约大学数据科学中心)
UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics
University of Helsinki(赫尔辛基大学);Zhongguancun Academy(中关村学院);University of Oxford(牛津大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Evaluating Time Series Models for Urban Wastewater Management: Predictive Performance, Model Complexity and Resilience
Berlin University of Applied Sciences and Technology(柏林应用科学与技术大学);Intelligent Embedded Systems Department, University of Duisburg-Essen(智能嵌入式系统系,杜伊斯堡- Essen大学);Einstein Center Digital Future(爱因斯坦数字未来研究中心)
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
School of Architecture and Urban Planning, Shenzhen University(深圳大学建筑与城市规划学院);Shenzhen Key Laboratory of Urban Spatial Information and Intelligent Modeling(深圳市城市空间信息与智能建模重点实验室);Department of Urban Planning and Design, The University of Hong Kong(香港大学城市规划与设计系)
UrbanGraphEmbeddings: Learning and Evaluating Spatially Grounded Multimodal Embeddings for Urban Science
National University of Singapore Department of Architecture Singapore;The Chinese University of Hong Kong Dept of Systems Eng. \& Eng. Mgmt. China;Singapore Management University School of Computing \& Info. Systems Singapore;National University of Singapore;Department of Architecture;The Chinese University of Hong Kong;Singapore Management University
Evaluating the Generalization Ability of Spatiotemporal Model in Urban Scenario
SUSTech-UTokyo Joint Research Center on Super Smart City(南方科技大学-东京大学超级智慧城市联合研究中心);Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology(南方科技大学可信自主系统研究院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Center for Spatial Information Science, University of Tokyo(东京大学空间信息科学中心)