精度赋能，过量分散：在语言模型中动态注入知识的视觉问答
Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
AI总结：
本文面向KBVQA，在OFA中利用动态三元组抽取按问题需求注入知识图谱上下文，实验显示其在三个数据集上平均Exact Match提升4.75%，并具备良好泛化性。
摘要：
在多模态任务领域，视觉问答（Visual Question Answering，VQA）通过回答基于视觉内容的自然语言问题而发挥关键作用。基于知识的视觉问答（Knowledge-Based Visual Question Answering，KBVQA）通过在图像之外加入外部知识来回答问题，从而推进了这一概念。我们提出一种用于KBVQA的方法，对现有的视觉-语言Transformer编码器-解码器（OFA）模型进行增强。我们的主要贡献在于：采用动态三元组抽取方法，通过纳入从知识图谱中抽取的相关外部知识来增强问题。我们根据回答问题的需求，提供数量灵活的知识图谱三元组作为上下文。我们的知识增强模型在三个不同的KBVQA数据集上，Exact Match Score较当前最先进水平平均提升4.75%。通过实验和分析，我们证明，与提供固定数量三元组相比，为每个问题提供可变数量三元组能够提升语言模型的推理能力；即使对于近期的大型语言模型，这一点也得到体现。此外，我们通过展示该模型仅经直接微调即可在小型数据集上取得超越SOTA的性能，突出了其泛化能力。
英文摘要：
In the realm of multimodal tasks, Visual Question Answering (VQA) plays a crucial role by addressing natural language questions grounded in visual content. Knowledge-Based Visual Question Answering (KBVQA) advances this concept by adding external knowledge along with images to respond to questions. We introduce an approach for KBVQA, augmenting the existing vision-language transformer encoder-decoder (OFA) model. Our main contribution involves enhancing questions by incorporating relevant external knowledge extracted from knowledge graphs, using a dynamic triple extraction method. We supply a flexible number of triples from the knowledge graph as context, tailored to meet the requirements for answering the question. Our model, enriched with knowledge, demonstrates an average improvement of 4.75\% in Exact Match Score over the state-of-the-art on three different KBVQA datasets. Through experiments and analysis, we demonstrate that furnishing variable triples for each question improves the reasoning capabilities of the language model in contrast to supplying a fixed number of triples. This is illustrated even for recent large language models. Additionally, we highlight the model's generalization capability by showcasing its SOTA-beating performance on a small dataset, achieved through straightforward fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
