利用视觉-语言模型实现建成环境的零样本标注（愿景论文）
Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)
AI总结：
本文提出利用视觉-语言模型结合预分割提示策略，从卫星图像中零样本标注停止线和凸起平台等城市特征，将交并比准确率从接近零提升至近40%，为建成环境自动标注研究提供新方向。
摘要：
公平的城市交通应用需要建成环境的高保真数字表示：不仅是街道和人行道，还包括自行车道、有标记和无标记的交叉口、路缘坡道和切口、障碍物、交通信号、标识、道路标线、坑洞等。直接检查和人工标注在大规模场景下成本过高。传统机器学习方法需要大量带标注的训练数据才能获得足够性能。在本文中，我们将视觉-语言模型视为一种从卫星图像中标注多样化城市特征的机制，从而减少对人工标注的依赖以生成大型训练集。尽管这些模型在描述从人类视角拍摄的图像中的常见物体方面取得了令人瞩目的成果，但它们的训练集不太可能包含建成环境中冷门特征的强信号，因此它们在这些场景下的性能尚不明确。我们展示了一个概念验证，结合了最先进的视觉-语言模型和一种提示策略的变体，该策略要求模型独立于原始图像来考虑分割后的元素。在两个城市特征——停止线和凸起平台——上的实验表明，虽然直接零样本提示几乎无法正确标注任何图像，但预分割策略能够以接近40%的交并比准确率标注图像。我们描述了这些结果如何为一个新的研究议程提供信息，该议程旨在实现建成环境的自动标注，以在广泛规模和多样化环境中改善公平性、可达性和安全性。
英文摘要：
Equitable urban transportation applications require high-fidelity digital representations of the built environment: not just streets and sidewalks, but bike lanes, marked and unmarked crossings, curb ramps and cuts, obstructions, traffic signals, signage, street markings, potholes, and more. Direct inspections and manual annotations are prohibitively expensive at scale. Conventional machine learning methods require substantial annotated training data for adequate performance. In this paper, we consider vision language models as a mechanism for annotating diverse urban features from satellite images, reducing the dependence on human annotation to produce large training sets. While these models have achieved impressive results in describing common objects in images captured from a human perspective, their training sets are less likely to include strong signals for esoteric features in the built environment, and their performance in these settings is therefore unclear. We demonstrate proof-of-concept combining a state-of-the-art vision language model and variants of a prompting strategy that asks the model to consider segmented elements independently of the original image. Experiments on two urban features -- stop lines and raised tables -- show that while direct zero-shot prompting correctly annotates nearly zero images, the pre-segmentation strategies can annotate images with near 40% intersection-over-union accuracy. We describe how these results inform a new research agenda in automatic annotation of the built environment to improve equity, accessibility, and safety at broad scale and in diverse environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
