arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.00871cs.CVcs.AI
收藏

城市感知中的视觉语言模型基准应具备可靠性意识且可协商

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

Rashid Mushkani

Rashid Mushkani

AI总结:

本文提出,用于城市感知的视觉语言模型基准应将分歧和弃权视为测量结果,报告标注者间信度,并将标签空间和评分策略视为可协商的产物。

摘要:

视觉语言模型(VLM)越来越多地用于生成街景图像的结构化描述,用于街道景观审计、制图和公众咨询等任务。这些用途将可观察属性与评估类别相结合,而人类目标往往是带有分歧和明确不回答的判断分布。本文认为,为城市感知建立VLM基准应将分歧和弃权视为测量结果,报告标注者间信度以及模型对齐度,并在输出旨在为城市治理提供信息时,将标签空间和评分策略视为可协商的产物。我们基于一个由来自七个社区组织的12名参与者对100个蒙特利尔街景进行30个维度标注的基准,以及对七个VLM的确定性零样本评估来论证这一观点。在各个维度上,模型与人类共识的一致性随维度层面的人类信度共同变化,而对于评估维度“总体印象”,模型和标注者表现出分布不匹配,包括“不适用”的不同比率。最后,我们为基准创建者、模型开发者和机构提出了行动建议,以使不确定性和基准假设在评估报告中可见。

英文摘要:

Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑