Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.
UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City
Shanghai Jiao Tong University(上海交通大学);National University of Singapore(新加坡国立大学);Meituan(美团);The Chinese University of Hong Kong(香港中文大学);Shanghai University(上海大学);University of Oxford(牛津大学)
UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics
University of Helsinki(赫尔辛基大学);Zhongguancun Academy(中关村学院);University of Oxford(牛津大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
School of Geospatial Engineering and Science(测绘工程与科学学院);Sun Yat-sen University(中山大学);College of Electronic Science(电子科学学院);National University of Defense Technology(国防科技大学);Department of Civil and Environmental Engineering(土木与环境工程系);Norwegian University of Science and Technology(挪威科技大学);National Geomatics Center of China(中国国家测绘地理信息中心)
Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)(信息工程测绘遥感国家重点实验室);Wuhan University(武汉大学);Hubei Luojia Laboratory(湖北珞珈实验室)
CityRiSE: Reasoning Urban Socio-Economic Status in Large Vision-Language Models via Reinforcement Learning
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)信息中心);Department of Electronic Engineering, BNRist, Tsinghua University(清华大学电子工程系)
Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda
ALGORITMI Research Centre/LASI(ALGORITMI研究机构/LASI);University of Minho(明霍大学);Logimade(Logimade公司);INESC TEC(INESC TEC研究机构);Ai, School of Technology(2Ai技术学院);IPCA
CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、北京理工大学、清华大学);School of Electronic and Information Engineering, Beijing Jiaotong University(电子信息工程学院、北京交通大学);Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院、清华大学)
Cross-View Urban Sensing: Mapping Subjective Streetscape Perception via AlphaEarth Embeddings and Urban Context
School of Geospatial Engineering and Science, Sun Yat-sen University(中山大学地理空间工程学院);Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai)(南方海洋科学与工程广东省实验室(珠海));Key Laboratory of Comprehensive Observation of Polar Environment (Sun Yat-sen University), Ministry of Education(教育部极地环境综合观测重点实验室(中山大学))
Learning Street View Representations with Spatiotemporal Contrast
Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University(遥感与地理信息系统研究所,地球与空间科学学院,北京大学);Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology(土木与环境工程系,香港科学与技术大学);SEAI Lab, Department of Geography and the Environment, University of Texas at Austin(SEAI实验室,地理与环境系,德克萨斯大学奥斯汀分校)
OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);PhiGent Robotics(鉴智机器人);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学)
CORP: A Multi-Modal Dataset for Campus-Oriented Roadside Perception Tasks
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合国家科学中心人工智能研究院);University of Science and Technology of China(中国科学技术大学)
EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racing
Skolkovo Institute of Science, Intelligent Space Robotics Laboratory, Center for Engineering Systems and Sciences(斯科尔科沃科学技术学院,智能空间机器人实验室,工程系统与科学中心);Khalifa University, KUCARS-KU Center for Autonomous Robotic Systems, Department of Computer Science(哈利法大学,KUCARS-KU自主机器人系统中心,计算机科学系);Beijing Institute of Technology, Beijing Key Laboratory of UAV Autonomous Control(北京理工大学,北京无人机自主控制重点实验室)
CSNR and JMIM Based Spectral Band Selection for Reducing Metamerism in Urban Driving
School of Engineering, University of Galway, Ireland(Galway大学工程学院,爱尔兰);Ryan Institute, University of Galway, Ireland(Galway大学Ryan学院,爱尔兰);Valeo Vision Systems, Tuam, Ireland(爱尔兰Tuam市Valeo Vision Systems公司)
Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与 Mobility学院);State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University(清华大学智能绿色车辆与交通全国重点实验室)
PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes
Zhejiang University, China(浙江大学);Autonomous Vision Group (AVG) at the University of Tübingen and Tübingen AI Center, Germany(图宾根大学自主视觉组(AVG)及图宾根人工智能中心)
It is not always greener on the other side: Greenery perception across demographics and personalities in multiple cities
Future Cities Lab Global, Singapore-ETH Centre(未来城市实验室全球,新加坡-ETH中心);Department of Architecture and Civil Engineering, City University of Hong Kong(香港城市大学建筑与土木工程系);Digital Geography Lab, Department of Geosciences and Geography, University of Helsinki(赫尔辛基大学地理科学与地理系数字地理实验室);Department of Architecture, National University of Singapore(新加坡国立大学建筑系);Department of Real Estate, National University of Singapore(新加坡国立大学房地产系)
PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors
IIIS, Tsinghua University(清华大学信息科学技术学院(简称:姚期智讲座教授信息技术科学学院));University of Science and Technology Beijing(北京科技大学);University of Southern California(南加州大学);Shanghai Qi Zhi Institute(上海期智研究院)
Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities
University of Liverpool(利物浦大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);JITRI(江苏省产业技术研究院);Nantong University(南通大学)