推进对AR生成场景的理解与评估：视觉语言模型的闪光与失误
Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble
AI总结：
本研究使用DiverseAR数据集评估了GPT、Gemini和Claude三种视觉语言模型在识别和描述AR生成场景方面的能力，发现其感知真阳性率高达93%，但在处理无缝整合内容时存在局限，突显了影响模型性能的关键因素。
摘要：
增强现实（AR）通过整合虚拟内容来增强真实世界，但确保AR体验的质量、可用性和安全性面临重大挑战。视觉语言模型（VLMs）能否为AR生成场景的自动化评估提供解决方案？在本研究中，我们评估了三种最先进的商业VLM——GPT、Gemini和Claude——在识别和描述AR场景方面的能力。为此，我们使用了DiverseAR，这是首个专门设计用于评估VLM分析各种AR场景复杂度下虚拟内容能力的AR数据集。我们的研究结果表明，VLM通常能够感知和描述AR场景，感知的真阳性率（TPR）高达93%，描述的TPR达71%。虽然它们擅长识别明显的虚拟物体，如发光的苹果，但在面对无缝整合的内容时却显得力不从心，例如带有逼真阴影的虚拟锅。我们的结果突显了VLM在理解AR场景方面的优势和局限性。我们确定了影响VLM性能的关键因素，包括虚拟内容放置、渲染质量和物理合理性。本研究强调了VLM作为评估AR体验质量工具的潜力。
英文摘要：
Augmented Reality (AR) enhances the real world by integrating virtual content, yet ensuring the quality, usability, and safety of AR experiences presents significant challenges. Could Vision-Language Models (VLMs) offer a solution for the automated evaluation of AR-generated scenes? Could Vision-Language Models (VLMs) offer a solution for the automated evaluation of AR-generated scenes? In this study, we evaluate the capabilities of three state-of-the-art commercial VLMs -- GPT, Gemini, and Claude -- in identifying and describing AR scenes. For this purpose, we use DiverseAR, the first AR dataset specifically designed to assess VLMs' ability to analyze virtual content across a wide range of AR scene complexities. Our findings demonstrate that VLMs are generally capable of perceiving and describing AR scenes, achieving a True Positive Rate (TPR) of up to 93% for perception and 71% for description. While they excel at identifying obvious virtual objects, such as a glowing apple, they struggle when faced with seamlessly integrated content, such as a virtual pot with realistic shadows. Our results highlight both the strengths and the limitations of VLMs in understanding AR scenarios. We identify key factors affecting VLM performance, including virtual content placement, rendering quality, and physical plausibility. This study underscores the potential of VLMs as tools for evaluating the quality of AR experiences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
