基于混合马尔可夫逻辑网络的视觉字幕生成可解释性研究
On Explaining Visual Captioning with Hybrid Markov Logic Networks
AI总结：
针对深度神经网络视觉字幕生成的可解释性难题，提出基于混合马尔可夫逻辑网络的解释框架，通过训练实例分布偏移量化影响样本，实验验证了方法的可解释性并可用于模型比较。
摘要：
深度神经网络（DNNs）已在图像字幕生成等多模态任务中取得巨大进展。然而，解释/解读这些模型如何整合视觉信息、语言信息与知识表征以生成有意义的字幕，仍是一项具有挑战性的难题。衡量模型性能的标准指标通常依赖于将生成的字幕与人工撰写的字幕进行对比，这可能无法让用户深入理解这种信息整合过程。在本研究中，我们基于混合马尔可夫逻辑网络（Hybrid Markov Logic Networks, HMLNs）——一种可结合符号规则与实值函数的语言——开发了一种易于解读的新型解释框架，我们在该框架中假设训练数据中的相关示例可能对观测到的字幕生成过程产生影响。为此，我们学习训练实例上的HMLN分布，并推断在以生成样本为条件时这些实例上的分布偏移，从而得以量化哪些示例可能是生成观测字幕的更丰富信息来源。我们使用Amazon Mechanical Turk，针对多个最先进的字幕生成模型生成的字幕开展实验，结果验证了我们的解释方法的可解读性，同时也让我们能够从可解释性维度对这些模型进行比较。
英文摘要：
Deep Neural Networks (DNNs) have made tremendous progress in multimodal tasks such as image captioning. However, explaining/interpreting how these models integrate visual information, language information and knowledge representation to generate meaningful captions remains a challenging problem. Standard metrics to measure performance typically rely on comparing generated captions with human-written ones that may not provide a user with a deep insights into this integration. In this work, we develop a novel explanation framework that is easily interpretable based on Hybrid Markov Logic Networks (HMLNs) - a language that can combine symbolic rules with real-valued functions - where we hypothesize how relevant examples from the training data could have influenced the generation of the observed caption. To do this, we learn a HMLN distribution over the training instances and infer the shift in distributions over these instances when we condition on the generated sample which allows us to quantify which examples may have been a source of richer information to generate the observed caption. Our experiments on captions generated for several state-of-the-art captioning models using Amazon Mechanical Turk illustrate the interpretability of our explanations, and allow us to compare these models along the dimension of explainability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
