arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.02155cs.CV
收藏

S3C:通过自批判学习的半监督VQA自然语言解释

S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning

Northwestern Polytechnical University(西北工业大学);University of Adelaide(阿德莱德大学)

Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, Qi Wu

AI总结:

提出S3C半监督框架,通过自批判学习与回答奖励提升VQA自然语言解释的逻辑一致性,无需人工标注,在两个数据集上达到最先进性能。

摘要:

VQA自然语言解释(VQA-NLE)任务旨在以自然语言解释VQA模型的决策过程。与传统的注意力或梯度分析不同,自由文本理由更易于理解并获得用户的信任。现有方法大多使用事后或自合理化模型来获得看似合理的解释。然而,这些框架受到以下挑战的制约:1)推理过程无法被忠实地响应,并遭受逻辑不一致的问题。2)人工标注的解释收集起来昂贵且耗时。在本文中,我们提出了一种新的通过自批判学习的半监督VQA-NLE(S3C),该方法通过回答奖励来评估候选解释,以提高答案与理由之间的逻辑一致性。借助半监督学习框架,S3C可以从大量无需人工标注解释的样本中获益。大量的自动评估和人工评估均证明了我们方法的有效性。同时,该框架在两个VQA-NLE数据集上取得了新的最先进性能。

英文摘要:

VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language. Unlike traditional attention or gradient analysis, free-text rationales can be easier to understand and gain users' trust. Existing methods mostly use post-hoc or self-rationalization models to obtain a plausible explanation. However, these frameworks are bottlenecked by the following challenges: 1) the reasoning process cannot be faithfully responded to and suffer from the problem of logical inconsistency. 2) Human-annotated explanations are expensive and time-consuming to collect. In this paper, we propose a new Semi-Supervised VQA-NLE via Self-Critical Learning (S3C), which evaluates the candidate explanations by answering rewards to improve the logical consistency between answers and rationales. With a semi-supervised learning framework, the S3C can benefit from a tremendous amount of samples without human-annotated explanations. A large number of automatic measures and human evaluations all show the effectiveness of our method. Meanwhile, the framework achieves a new state-of-the-art performance on the two VQA-NLE datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑