一个让其持续学习的VLM：面向无数据持续视觉问答的生成与平衡
One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering
AI总结：
针对持续VQA中的灾难性遗忘与旧数据不可用问题，本文提出无数据GaB，利用单一VLM在新任务数据上生成旧任务伪排练样本，并通过元统计或聚类平衡分布，在两个基准上达到可比有数据方法的性能。
摘要：
视觉-语言模型（VLM）通过利用网络规模的多模态数据集，在视觉问答（VQA）任务中展现出显著前景。然而，这些模型在适应新任务时往往因灾难性遗忘而难以进行持续学习。作为缓解灾难性遗忘的有效补救措施，排练策略在学习新任务时使用过去任务的数据。但这种策略需要存储过去数据，而由于硬件限制或隐私问题，这可能并不可行。在本研究中，我们提出首个无数据方法，该方法利用VLM自身的语言生成能力而非依赖外部模型，来生成伪排练数据以解决持续VQA问题。我们的方案名为GaB，通过在新任务数据上提出先前任务的问题来生成伪排练数据。然而，尽管该方法有效，但由于训练数据有限且具有任务特定性，所生成问题的分布会偏向最常提出的问题。为缓解这一问题，我们引入了一个伪排练平衡模块，该模块使用问题元统计信息或无监督聚类方法，使生成的数据与真实数据分布对齐。我们在两个近期基准上评估所提出的方法，即VQACL-VQAv2和CLOVE-function基准。GaB在跨不断演化任务保持VQA性能方面显著优于所有无数据基线，同时与能够访问过去数据的方法表现相当。
英文摘要：
Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However, these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting, rehearsal strategy uses the data of past tasks upon learning new task. However, such strategy incurs the need of storing past data, which might not be feasible due to hardware constraints or privacy concerns. In this work, we propose the first data-free method that leverages the language generation capability of a VLM, instead of relying on external models, to produce pseudo-rehearsal data for addressing continual VQA. Our proposal, named as GaB, generates pseudo-rehearsal data by posing previous task questions on new task data. Yet, despite being effective, the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue, we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks, \ie VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks, while being on-par with methods with access to the past data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
