你为何出错？面向3D物体语言接地的反事实解释
Why Are You Wrong? Counterfactual Explanations for Language Grounding with 3D Objects
AI总结：
本文针对3D物体语言接地中模型错误预测难以解释的问题，提出通过生成保持原描述结构且语义相似的反事实示例来揭示描述弱点与模型偏差，并在ShapeTalk数据集和三个模型上验证了其有效性。
摘要：
自然语言与几何形状的结合是一个新兴研究领域，在机器人学和语言辅助设计中具有多种应用。该领域的一项关键任务是物体指称识别，即在给定目标文本描述的情况下选择一个3D物体。语言描述的多样性和3D物体的空间关系使这一任务变得复杂，从而增加了更好地理解该领域神经网络模型行为的需求。然而，目前在这一方面的研究十分有限。具体而言，当模型尽管被提供了看似正确的物体描述却仍做出错误预测时，实践者会疑惑：“模型为什么错了？”在本工作中，我们提出了一种通过生成反事实示例来回答这一问题的方法。我们的方法以一个被错误分类的样本为输入，该样本包含两个物体和一段文本描述，并生成一个替代但相似的表述，该表述本应使模型做出正确预测。我们使用ShapeTalk数据集的数据以及三个不同的模型对我们的方法进行了评估。我们的反事实示例保持了原始描述的结构，在语义上相似且有意义。它们揭示了描述中的弱点、模型偏差，并增强了对模型行为的理解。这些见解有助于实践者更好地与系统交互，也有助于工程师改进模型。
英文摘要：
Combining natural language and geometric shapes is an emerging research area with multiple applications in robotics and language-assisted design. A crucial task in this domain is object referent identification, which involves selecting a 3D object given a textual description of the target. Variability in language descriptions and spatial relationships of 3D objects makes this a complex task, increasing the need to better understand the behavior of neural network models in this domain. However, limited research has been conducted in this area. Specifically, when a model makes an incorrect prediction despite being provided with a seemingly correct object description, practitioners are left wondering: "Why is the model wrong?". In this work, we present a method answering this question by generating counterfactual examples. Our method takes a misclassified sample, which includes two objects and a text description, and generates an alternative yet similar formulation that would have resulted in a correct prediction by the model. We have evaluated our approach with data from the ShapeTalk dataset along with three distinct models. Our counterfactual examples maintain the structure of the original description, are semantically similar and meaningful. They reveal weaknesses in the description, model bias and enhance the understanding of the models behavior. Theses insights help practitioners to better interact with systems as well as engineers to improve models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
