探索视觉语言模型检测HTML5 <canvas>应用中视觉缺陷的能力
Exploring the Capabilities of Vision-Language Models to Detect Visual Bugs in HTML5 <canvas> Applications
AI总结：
针对使用程序化图形的HTML5 <canvas>应用难以检测视觉缺陷的问题，研究提出利用视觉语言模型，结合应用说明、缺陷类型描述及无缺陷截图作为上下文，实现最高100%的单应用视觉缺陷检测准确率。
摘要：
超文本标记语言5（HTML5）<canvas>可用于创建以视觉为中心的Web应用。但与传统Web应用不同，HTML5 <canvas>应用将对象渲染到<canvas>位图上，不会在文档对象模型（DOM）中表示这些对象。<canvas>位图的预期视觉输出与实际视觉输出之间的不匹配被称为视觉缺陷。由于<canvas>应用以视觉为核心，检测视觉缺陷十分重要，因为这类缺陷可能导致<canvas>应用完全无法使用。正如我们在先前工作中展示的，基于资源的图形（Asset-Based graphics）可以为视觉测试预言机提供基准真值。但许多<canvas>应用采用程序化方式生成图形。本文中，我们研究了如何在同样使用程序化图形（Procedural graphics）的<canvas>应用中检测视觉缺陷。我们重点探索了视觉语言模型（Vision-Language Models, VLMs）自动检测视觉缺陷的潜力。该方法无需定义精确的视觉测试预言机，而是可以将应用预期功能的相关信息（即上下文）与截图一同作为输入提供给VLM。为评估该方法，我们构建了一个数据集，包含来自20个<canvas>应用的、覆盖四种视觉缺陷类型（布局、渲染、外观、状态）的80张注入缺陷的截图，以及20张无缺陷截图。我们使用当前最先进的VLM开展实验，采用多种文本与图像上下文组合来描述每个应用的预期功能。结果表明，通过提供应用README文件、视觉缺陷类型说明以及无缺陷截图作为上下文，利用VLM检测视觉缺陷的单应用准确率最高可达100%。
英文摘要：
The HyperText Markup Language 5 (HTML5) <canvas> is useful for creating visual-centric web applications. However, unlike traditional web applications, HTML5 <canvas> applications render objects onto the <canvas> bitmap without representing them in the Document Object Model (DOM). Mismatches between the expected and actual visual output of the <canvas> bitmap are termed visual bugs. Due to the visual-centric nature of <canvas> applications, visual bugs are important to detect because such bugs can render a <canvas> application useless. As we showed in prior work, Asset-Based graphics can provide the ground truth for a visual test oracle. However, many <canvas> applications procedurally generate their graphics. In this paper, we investigate how to detect visual bugs in <canvas> applications that use Procedural graphics as well. In particular, we explore the potential of Vision-Language Models (VLMs) to automatically detect visual bugs. Instead of defining an exact visual test oracle, information about the application's expected functionality (the context) can be provided with the screenshot as input to the VLM. To evaluate this approach, we constructed a dataset containing 80 bug-injected screenshots across four visual bug types (Layout, Rendering, Appearance, and State) plus 20 bug-free screenshots from 20 <canvas> applications. We ran experiments with a state-of-the-art VLM using several combinations of text and image context to describe each application's expected functionality. Our results show that by providing the application README(s), a description of visual bug types, and a bug-free screenshot as context, VLMs can be leveraged to detect visual bugs with up to 100% per-application accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
