arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.02411cs.CLcs.AIcs.CV
收藏

知识在视觉语言学习中的贡献:任务与挑战综述

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

National Technical University of Athens(雅典国家技术大学);School of Electrical and Computer Engineering(电气与计算机工程学院);AILS Laboratory(AILS实验室)

Maria Lymperaiou, Giorgos Stamou

AI总结:

本文综述了利用知识图谱和大语言模型等外部知识源弥补视觉语言模型泛化不足的混合方法,分析了相关任务并对知识来源进行分类,探讨了KG与LLM的选择困境及未来影响。

摘要:

视觉语言(VL)学习的最新进展推动了多种模型和技术的发展,这些模型和技术提供了若干令人印象深刻的实现,目前能够解决各种需要视觉与语言协作的任务。当前用于VL预训练的数据集仅包含有限数量的视觉和语言知识,因此显著限制了许多VL模型的泛化能力。外部知识源,如知识图谱(KGs)和大语言模型(LLMs),能够通过填补缺失的知识来弥补这种泛化差距,从而催生了混合架构的出现。在本综述中,我们分析了从此类混合方法中受益的任务。此外,我们对现有的知识来源和类型进行了分类,进而讨论了KG与LLM之间的两难选择及其对未来混合方法的潜在影响。

英文摘要:

Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑