arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.09352cs.LGcs.MMeess.IV
收藏

PAL：面向多模态类别增量学习的缺失模态提示分析学习

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

National University of Singapore（新加坡国立大学）；National University of Defense Technology（国防科技大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（华南理工大学伍舜德智能工程学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）

Xianghu Yue, Yiming Chen, Xueyi Zhang, Xiaoxue Gao, Mengling Feng, Mingrui Lao, Huiping Zhuang, Haizhou Li

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态类别增量学习中的模态缺失导致遗忘加剧的问题，提出无范例框架PAL，通过模态专属提示补全信息、重构为递归最小二乘任务求解，在多数据集上实现更优性能与抗遗忘能力。

摘要：

多模态类别增量学习（MMCIL）旨在利用音视频、图文对这类多模态数据，使模型能在一系列任务中持续学习，同时缓解遗忘问题。现有研究主要聚焦于MMCIL中多模态信息的整合与利用，但仍存在一个关键挑战：增量学习阶段的模态缺失问题。这一疏漏会加剧严重遗忘，大幅损害模型性能。为填补这一空白，我们提出PAL，这是一款专为缺失模态场景下的MMCIL设计的新型无范例框架。具体而言，我们设计了模态专属提示来补偿缺失信息，帮助模型维持数据的整体表征。在此基础上，我们将MMCIL问题重构为递归最小二乘任务，给出解析线性解。基于这些设计，PAL不仅缓解了分析学习固有的欠拟合局限，还能保留缺失模态数据的整体表征，在多种多模态增量场景中实现更优性能且遗忘更少。大量实验表明，在UPMC-Food101、N24News等多个数据集上，PAL的表现显著优于同类竞争方法，展现出其对模态缺失的鲁棒性，以及维持高增量准确率的抗遗忘能力。

英文摘要：

Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a sequence of tasks while mitigating forgetting. While existing studies primarily focus on the integration and utilization of multi-modal information for MMCIL, a critical challenge remains: the issue of missing modalities during incremental learning phases. This oversight can exacerbate severe forgetting and significantly impair model performance. To bridge this gap, we propose PAL, a novel exemplar-free framework tailored to MMCIL under missing-modality scenarios. Concretely, we devise modality-specific prompts to compensate for missing information, facilitating the model to maintain a holistic representation of the data. On this foundation, we reformulate the MMCIL problem into a Recursive Least-Squares task, delivering an analytical linear solution. Building upon these, PAL not only alleviates the inherent under-fitting limitation in analytic learning but also preserves the holistic representation of missing-modality data, achieving superior performance with less forgetting across various multi-modal incremental scenarios. Extensive experiments demonstrate that PAL significantly outperforms competitive methods across various datasets, including UPMC-Food101 and N24News, showcasing its robustness towards modality absence and its anti-forgetting ability to maintain high incremental accuracy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑