通过HSIC-瓶颈正交化与等角嵌入实现持续学习目标
Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding
AI总结：
本文针对无样本缓冲区且模型规模恒定的严格持续学习场景，提出结合HSIC-Bottleneck正交化与等角嵌入的方法，通过减少参数覆盖和改善决策边界适应来缓解灾难性遗忘，并在实验中展现出有竞争力的准确率。
摘要：
深度神经网络在顺序任务上训练时容易遭受灾难性遗忘。各种持续学习（CL）方法通常依赖样本缓冲区或/和网络扩展来平衡模型稳定性与可塑性，然而由于隐私和内存方面的顾虑，这损害了它们的实用价值。相反，本文考虑一种严格但现实的设置：在顺序训练过程中，先前任务的训练数据不可用，且模型规模保持相对恒定。为了实现这些目标，我们提出了一种概念简单但有效的方法，将遗忘归因于逐层参数覆盖以及由此产生的决策边界畸变。这一方法通过两个关键组件的协同作用实现：HSIC-Bottleneck正交化（HBO）在正交空间中借助Hilbert-Schmidt独立性准则实现非覆盖式参数更新，而等角嵌入（EAE）利用预定义基向量增强新旧任务之间的决策边界适应能力。大量实验表明，我们的方法取得了有竞争力的准确率性能，甚至在零样本缓冲区和1.02倍基础模型规模下具有绝对优势。
英文摘要：
Deep neural networks are susceptible to catastrophic forgetting when trained on sequential tasks. Various continual learning (CL) methods often rely on exemplar buffers or/and network expansion for balancing model stability and plasticity, which, however, compromises their practical value due to privacy and memory concerns. Instead, this paper considers a strict yet realistic setting, where the training data from previous tasks is unavailable and the model size remains relatively constant during sequential training. To achieve such desiderata, we propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. This is achieved by the synergy between two key components: HSIC-Bottleneck Orthogonalization (HBO) implements non-overwritten parameter updates mediated by Hilbert-Schmidt independence criterion in an orthogonal space and EquiAngular Embedding (EAE) enhances decision boundary adaptation between old and new tasks with predefined basis vectors. Extensive experiments demonstrate that our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
