TinySubNets：一种高效且低容量的持续学习策略
TinySubNets: An efficient and low capacity continual learning strategy
AI总结：
本文提出TinySubNets（TSN）持续学习策略，通过结合不同稀疏度剪枝、自适应量化与权重共享，有效解决模型容量饱和问题，在准确率和容量利用率上均优于现有最先进方法。
摘要：
持续学习（CL）是近期机器学习研究中备受关注且高度相关的设定。在CL研究中，架构型和混合型策略由于能够随着新任务的呈现调整模型架构而尤为有效。然而，许多现有解决方案未能有效利用模型稀疏性，且由于对可用权重的低效使用容易导致容量饱和，从而限制了可学习任务的数量。本文提出TinySubNets（TSN），一种新颖的架构型CL策略，通过将不同稀疏度级别的剪枝、自适应量化和权重共享独特结合来解决上述问题。剪枝识别出能保持模型性能的权重子集，使相关性较低的权重可用于未来任务。自适应量化允许将单个权重分离为多个部分，并分配给不同任务。任务间的权重共享促进了对容量和任务相似性的利用，从而在模型准确率和容量之间实现更好的权衡。这些特性使TSN能够有效利用可用容量，增强知识迁移，并降低计算资源消耗。涉及常见基准CL数据集和场景的实验结果表明，我们提出的策略在准确率方面优于现有的最先进CL策略。此外，结果表明我们的策略显著提升了模型容量的利用率。代码发布于：https://github.com/lifelonglab/tinysubnets。
英文摘要：
Continual Learning (CL) is a highly relevant setting gaining traction in recent machine learning research. Among CL works, architectural and hybrid strategies are particularly effective due to their potential to adapt the model architecture as new tasks are presented. However, many existing solutions do not efficiently exploit model sparsity, and are prone to capacity saturation due to their inefficient use of available weights, which limits the number of learnable tasks. In this paper, we propose TinySubNets (TSN), a novel architectural CL strategy that addresses the issues through the unique combination of pruning with different sparsity levels, adaptive quantization, and weight sharing. Pruning identifies a subset of weights that preserve model performance, making less relevant weights available for future tasks. Adaptive quantization allows a single weight to be separated into multiple parts which can be assigned to different tasks. Weight sharing between tasks boosts the exploitation of capacity and task similarity, allowing for the identification of a better trade-off between model accuracy and capacity. These features allow TSN to efficiently leverage the available capacity, enhance knowledge transfer, and reduce computational resource consumption. Experimental results involving common benchmark CL datasets and scenarios show that our proposed strategy achieves better results in terms of accuracy than existing state-of-the-art CL strategies. Moreover, our strategy is shown to provide a significantly improved model capacity exploitation. Code released at: https://github.com/lifelonglab/tinysubnets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
