基于视觉语言模型的持续学习用于自动驾驶中的视觉问答
VLM-Assisted Continual learning for Visual Question Answering in Self-Driving
AI总结：
本文提出结合视觉语言模型与持续学习的方法，以提升自动驾驶中视觉问答系统的性能和可靠性。
摘要：
本文提出了一种新的方法，用于解决自动驾驶中的视觉问答（VQA）任务，通过将视觉语言模型（VLMs）与持续学习相结合。在自动驾驶中，VQA在使系统理解并推理解其周围环境方面起着至关重要的作用。然而，传统模型在依次暴露于新驾驶任务时，如感知、预测和规划，往往面临灾难性遗忘的问题，每个任务都需要不同的知识形式。为了解决这一挑战，我们提出了一种新的持续学习框架，结合VLMs与选择性记忆重放和知识蒸馏，并通过任务特定的投影层正则化来加强。知识蒸馏允许先前训练的模型充当“教师”以指导后续任务，从而最小化遗忘。同时，任务特定的投影层根据特征表示的分歧计算损失，确保学习的连续性，并减少任务之间的转移。在DriveLM数据集上的评估表明，我们的框架在各种指标上显示出显著的性能提升，提升幅度在20.11%到35.16%之间。这些结果突显了将持续学习与VLMs结合在增强自动驾驶中VQA系统鲁棒性和可靠性方面的有效性。我们将发布我们的源代码。
英文摘要：
In this paper, we propose a novel approach for solving the Visual Question Answering (VQA) task in autonomous driving by integrating Vision-Language Models (VLMs) with continual learning. In autonomous driving, VQA plays a vital role in enabling the system to understand and reason about its surroundings. However, traditional models often struggle with catastrophic forgetting when sequentially exposed to new driving tasks, such as perception, prediction, and planning, each requiring different forms of knowledge. To address this challenge, we present a novel continual learning framework that combines VLMs with selective memory replay and knowledge distillation, reinforced by task-specific projection layer regularization. The knowledge distillation allows a previously trained model to act as a "teacher" to guide the model through subsequent tasks, minimizing forgetting. Meanwhile, task-specific projection layers calculate the loss based on the divergence of feature representations, ensuring continuity in learning and reducing the shift between tasks. Evaluated on the DriveLM dataset, our framework shows substantial performance improvements, with gains ranging from 20.11% to 35.16% across various metrics. These results highlight the effectiveness of combining continual learning with VLMs in enhancing the resilience and reliability of VQA systems in autonomous driving. We will release our source code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
