基于搜索的自主水下机器人软件中视觉语言模型的蜕变测试

中文摘要 AI 辅助

我们的行业合作伙伴专注于多个领域工业系统的质量保证，包括海事系统，如水面船舶和自主水下机器人（AUR）。尽管视觉语言模型（VLM）在场景理解、图像描述和物体识别方面表现出强大的性能，但它们在运行于水下环境的AUR软件中的应用尚未得到充分探索。因此，在此背景下，评估VLM集成到AUR软件中的质量非常重要，因此需要自动化软件测试工具来评估其适用性并提高其可靠性。为此，我们提出了一种基于搜索的蜕变测试方法（MetaVLM），该方法识别水下图像上的最小变换集以诱导错误的模型预测，从而揭示VLM的失败。我们采用NSGA-II作为多目标搜索算法，并针对随机搜索基线，在开源VLM（BLIP和CLIP）上对其进行评估。结果展示了每个VLM在AUR软件系统背景下的优势和局限性。基于这些结果，我们为从事基于VLM的软件系统质量保证的软件工程实践者和研究人员总结了经验教训。

英文摘要

Our industry partner focuses on quality assurance for industrial systems across multiple domains, including maritime systems, such as overwater vessels and autonomous underwater robots (AURs). Despite the strong performance of vision-language models (VLMs) in scene understanding, image captioning, and object recognition, their use in AUR software operating in underwater environments is underexplored. Therefore, in this context, it is important to evaluate the quality of VLMs for integration into AUR software and, so, automated software testing tools are needed to assess their suitability and improve their dependability. To this end, we propose a search-based metamorphic testing approach (MetaVLM) that identifies a minimal set of transformations on underwater images to induce incorrect model predictions, thereby revealing VLM failures. We employ NSGA-II as a multi-objective search algorithm and evaluate it over open-source VLMs, BLIP and CLIP, against a random search baseline. Results demonstrate the strengths and limitations of each VLM in the context of AUR software systems. Based on the results, we derive lessons for software engineering practitioners and researchers working on quality assurance of VLM-based software systems.

发表机构

• Simula Research Laboratory（Simula 研究实验室）
• Oslo Metropolitan University（奥斯陆都市大学）
• Mondragon University（蒙德拉贡大学）
• National Institute of Informatics（国立情报学研究所）
• Group Research and Development, DNV AS（DNV AS 集团研发部）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

15 pages