Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments
Bordeaux INP - ENSEIRB-MATMECA(波尔多 INP - ENSEIRB-MATMECA);CAMaCS, University of Warwick(Warwick 大学 CAMaCS 实验室);Institute of Computer Science Polish Academy of Sciences(波兰科学院计算机科学研究所);University of Pisa(比萨大学)
Clea Rebillard, Julio Hurtado, Andrii Krutsylo, Lucia Passaro, Vincenzo Lomonaco
Learning continually from a stream of non-i.i.d. data is an open challenge in deep learning, even more so when working in resource-constrained environments such as embedded devices. Visual models that are continually updated through supervised learning are often prone to overfitting, catastrophic forgetting, and biased representations. On the other hand, large language models contain knowledge about multiple concepts and their relations, which can foster a more robust, informed and coherent learning process. This work proposes Continual Visual Mapping (CVM), an approach that continually ground vision representations to a knowledge space extracted from a fixed Language model. Specifically, CVM continually trains a small and efficient visual model to map its representations into a conceptual space established by a fixed Large Language Model. Due to their smaller nature, CVM can be used when directly adapting large visual pre-trained models is unfeasible due to computational or data constraints. CVM overcome state-of-the-art continual learning methods on five benchmarks and offers a promising avenue for addressing generalization capabilities in continual learning, even in computationally constrained devices.
A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);Nanyang Technological University, Singapore(新加坡南洋理工大学);University College London(伦敦大学学院)
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(厦门大学多媒体可信感知与高效计算教育部重点实验室);National University of Defense Technology(国防科技大学)
CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心);Department of Informatics, King’s College London(伦敦国王学院信息学院);The Alan Turing Institute(艾伦·图灵研究所);AI Centre, Department of Computer Science, UCL(UCL计算机科学系人工智能中心)
Neural Visual Decoding via Cognitive guided Adaptive Blurring and Information Constrained Alignment
Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);Department of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院)