VisMoDAl：用于评估与提升视觉-语言模型抗损坏鲁棒性的可视化分析
VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models
AI总结：
本文提出可视化分析框架VisMoDAl，用于评估视觉-语言模型在多种数据损坏下的鲁棒性并识别低表现样本，从而指导数据增强策略制定。
摘要：
视觉-语言（VL）模型因其理解多模态信息的能力，在各个关键领域展现出变革性潜力。然而，它们的性能在分布偏移下经常下降，因此评估并提升其面对实际应用中真实世界数据损坏时的鲁棒性至关重要。尽管VL基准数据集和数据增强（DA）方面的进展有助于鲁棒性评估与提升，但由于缺乏对模型行为的深入理解，以及探索数据模式需要专业知识和迭代努力，仍然存在诸多挑战。鉴于可视化在解释复杂模型和探索大规模数据方面取得的成就，理解各种数据损坏对VL模型的影响自然与可视化分析方法相契合。为应对这些挑战，我们引入了VisMoDAl，这是一个可视化分析框架，旨在评估VL模型面对各种损坏类型时的鲁棒性，并识别表现不佳的样本，以指导有效DA策略的制定。基于文献综述和专家讨论，VisMoDAl支持多层次分析，从检查特定损坏下的性能，到任务驱动的模型行为检查及相应数据切片分析。与传统工作不同，VisMoDAl使用户能够推理损坏对VL模型的影响，从而促进模型行为理解和DA策略制定。我们通过聚焦于图像描述任务中损坏鲁棒性的案例研究和定量评估，展示了该系统的实用性。
英文摘要：
Vision-language (VL) models have shown transformative potential across various critical domains due to their capability to comprehend multi-modal information. However, their performance frequently degrades under distribution shifts, making it crucial to assess and improve robustness against real-world data corruption encountered in practical applications. While advancements in VL benchmark datasets and data augmentation (DA) have contributed to robustness evaluation and improvement, there remain challenges due to a lack of in-depth comprehension of model behavior as well as the need for expertise and iterative efforts to explore data patterns. Given the achievement of visualization in explaining complex models and exploring large-scale data, understanding the impact of various data corruption on VL models aligns naturally with a visual analytics approach. To address these challenges, we introduce VisMoDAl, a visual analytics framework designed to evaluate VL model robustness against various corruption types and identify underperformed samples to guide the development of effective DA strategies. Grounded in the literature review and expert discussions, VisMoDAl supports multi-level analysis, ranging from examining performance under specific corruptions to task-driven inspection of model behavior and corresponding data slice. Unlike conventional works, VisMoDAl enables users to reason about the effects of corruption on VL models, facilitating both model behavior understanding and DA strategy formulation. The utility of our system is demonstrated through case studies and quantitative evaluations focused on corruption robustness in the image captioning task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
