可能会出什么错？发现并描述计算机视觉中的失败模式
What could go wrong? Discovering and describing failure modes in computer vision
AI总结：
本文提出通过自然语言预测和描述计算机视觉模型的潜在失败模式，形式化了基于语言的错误可解释性问题（LBEE），并提出联合视觉与语言嵌入空间的解决方案，以提升模型的安全性和可解释性。
摘要：
深度学习模型虽然有效，但也很脆弱。即使经过精心训练，它们在面对分布外样本时，其行为往往也难以预测。在这项工作中，我们的目标是提出一种简单而有效的解决方案，通过自然语言来预测和描述计算机视觉模型的潜在失败模式。给定一个预训练模型和一组样本，我们的目标是找到能够准确描述模型表现不佳的视觉条件的句子。为了研究这一重要课题并促进未来的研究，我们形式化了基于语言的错误可解释性问题，并提出了一组用于评估和比较不同方法的指标。我们提出的解决方案在联合视觉与语言嵌入空间中运行，能够通过语言描述来刻画由训练期间未见的物体或不利视觉条件等引起的模型失败。我们在不同任务上进行了实验，例如存在数据集偏差情况下的分类和未见环境中的语义分割，并表明所提出的方法能够分离出与特定错误原因相关的非平凡句子。我们希望我们的工作能帮助从业者更好地理解模型的行为，从而提高其整体安全性和可解释性。
英文摘要：
Deep learning models are effective, yet brittle. Even carefully trained, their behavior tends to be hard to predict when confronted with out-of-distribution samples. In this work, our goal is to propose a simple yet effective solution to predict and describe via natural language potential failure modes of computer vision models. Given a pretrained model and a set of samples, our aim is to find sentences that accurately describe the visual conditions in which the model underperforms. In order to study this important topic and foster future research on it, we formalize the problem of Language-Based Error Explainability (LBEE) and propose a set of metrics to evaluate and compare different methods for this task. We propose solutions that operate in a joint vision-and-language embedding space, and can characterize through language descriptions model failures caused, e.g., by objects unseen during training or adverse visual conditions. We experiment with different tasks, such as classification under the presence of dataset bias and semantic segmentation in unseen environments, and show that the proposed methodology isolates nontrivial sentences associated with specific error causes. We hope our work will help practitioners better understand the behavior of models, increasing their overall safety and interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
