arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.08279cs.AI
收藏

资源受限环境下持续学习将视觉概念映射到大语言模型

Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments

Bordeaux INP - ENSEIRB-MATMECA(波尔多 INP - ENSEIRB-MATMECA);CAMaCS, University of Warwick(Warwick 大学 CAMaCS 实验室);Institute of Computer Science Polish Academy of Sciences(波兰科学院计算机科学研究所);University of Pisa(比萨大学)

Clea Rebillard, Julio Hurtado, Andrii Krutsylo, Lucia Passaro, Vincenzo Lomonaco

AI总结:

针对资源受限环境下视觉持续学习易过拟合、遗忘的问题,提出CVM方法,将轻量视觉模型表征映射到固定大语言模型的概念空间,在5个基准上超越现有最优方法。

摘要:

从非独立同分布数据流中持续学习是深度学习领域的一个开放性挑战,在嵌入式设备等资源受限环境中开展这类学习的难度更大。通过监督学习持续更新的视觉模型往往容易出现过拟合、灾难性遗忘和表征有偏的问题。另一方面,大语言模型包含关于多种概念及其关系的知识,能够促进更鲁棒、更具信息性且更连贯的学习过程。本研究提出了持续视觉映射(Continual Visual Mapping,CVM)方法,该方法可将视觉表征持续锚定到从固定语言模型中提取的知识空间中。具体而言,CVM持续训练一个轻量高效的视觉模型,将其表征映射到由固定大语言模型构建的概念空间中。由于该视觉模型体量更小,当计算或数据约束导致直接适配大型视觉预训练模型不可行时,即可使用CVM。CVM在5个基准测试上的表现超越了现有最优持续学习方法,为提升持续学习的泛化能力(即便在计算受限设备中)提供了一条颇具前景的路径。

英文摘要:

Learning continually from a stream of non-i.i.d. data is an open challenge in deep learning, even more so when working in resource-constrained environments such as embedded devices. Visual models that are continually updated through supervised learning are often prone to overfitting, catastrophic forgetting, and biased representations. On the other hand, large language models contain knowledge about multiple concepts and their relations, which can foster a more robust, informed and coherent learning process. This work proposes Continual Visual Mapping (CVM), an approach that continually ground vision representations to a knowledge space extracted from a fixed Language model. Specifically, CVM continually trains a small and efficient visual model to map its representations into a conceptual space established by a fixed Large Language Model. Due to their smaller nature, CVM can be used when directly adapting large visual pre-trained models is unfeasible due to computational or data constraints. CVM overcome state-of-the-art continual learning methods on five benchmarks and offers a promising avenue for addressing generalization capabilities in continual learning, even in computationally constrained devices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑