arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.07993cs.CL
收藏

CoLLIE:基于图像嵌入的语言Grounding的持续学习

CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings

KTH Royal Institute of Technology(KTH皇家理工学院)

Gabriel Skantze, Bram Willemsen

摘要:

本文介绍了CoLLIE:一个简单,但有效的持续学习的语言是如何在视觉的基础模型。给定一个预先训练的多模态嵌入模型,其中语言和图像被投影在相同的语义空间中(在这种情况下,OpenAI的CLIP),CoLLIE学习一个转换函数,该函数在需要时调整语言嵌入以适应新的语言使用。这是通过预测需要应用的差向量以及该向量的缩放因子来完成的,使得仅在需要时应用调整。与传统的少样本学习不同,该模型不仅学习新的类和标签,还可以推广到类似的语言使用并利用语义组合。我们验证了该模型的性能上的两个不同的任务,确定的目标是指表达,它必须学习新的语言使用。结果表明,该模型可以有效地学习和推广,只有少数的例子,与模型的原始zero-shot性能的干扰很小。

英文摘要:

This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same semantic space (in this case CLIP by OpenAI), CoLLIE learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. This is done by predicting the difference vector that needs to be applied, as well as a scaling factor for this vector, so that the adjustment is only applied when needed. Unlike traditional few-shot learning, the model does not just learn new classes and labels, but can also generalize to similar language use and leverage semantic compositionality. We verify the model's performance on two different tasks of identifying the targets of referring expressions, where it has to learn new language use. The results show that the model can efficiently learn and generalize from only a few examples, with little interference with the model's original zero-shot performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑