面向全二值网络中类增量学习的经验回放研究
Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks
AI总结：
本文研究全二值神经网络中的类增量学习，通过重新设计训练流程、损失平衡、半监督预训练特征提取器以及比较潜在回放与原生回放，在CIFAR100和CORE50上取得与更大实值模型相当或更优的性能。
摘要：
二值神经网络（BNNs）是一种在超低功耗边缘设备上实现人工神经网络（ANN）的有前景的方法。此类设备可能在高度动态的环境中计算数据，其中用于推理的目标类别可能演变，甚至可能出现新类别，从而需要持续学习。类增量学习（CIL）是分类问题中一种常见的持续学习类型，但在BNNs背景下很少被研究。此外，现有的大多数BNN模型并非完全二值，因为它们在输入、输出和批归一化处需要若干实值网络层。本文更进一步，通过四项主要贡献在全二值化神经网络（FBNNs）中实现类增量学习。首先，我们重新审视了适用于CIL的FBNN设计及其训练流程。其次，我们探索了损失平衡，这是一种在旧类别与当前类别性能之间进行权衡的方法。第三，我们提出了一种半监督方法，用于预训练FBNN的特征提取器以获得可迁移表示。第四，对两种传统CIL方法，即潜在回放和原生回放，进行了全面比较。这些贡献首先在CIFAR100数据集上进行了示例验证，随后扩展到CORE50持续学习基准。基于我们3Mb FBNN在CORE50上的最终结果表明，其性能与传统的实值更大神经网络模型相当甚至更优。
英文摘要：
Binary Neural Networks (BNNs) are a promising approach to enable Artificial Neural Network (ANN) implementation on ultra-low power edge devices. Such devices may compute data in highly dynamic environments, in which the classes targeted for inference can evolve or even novel classes may arise, requiring continual learning. Class Incremental Learning (CIL) is a common type of continual learning for classification problems, that has been scarcely addressed in the context of BNNs. Furthermore, most of existing BNNs models are not fully binary, as they require several real-valued network layers, at the input, the output, and for batch normalization. This paper goes a step further, enabling class incremental learning in Fully-Binarized NNs (FBNNs) through four main contributions. We firstly revisit the FBNN design and its training procedure that is suitable to CIL. Secondly, we explore loss balancing, a method to trade-off the performance of past and current classes. Thirdly, we propose a semi-supervised method to pre-train the feature extractor of the FBNN for transferable representations. Fourthly, two conventional CIL methods, \ie, Latent and Native replay, are thoroughly compared. These contributions are exemplified first on the CIFAR100 dataset, before being scaled up to address the CORE50 continual learning benchmark. The final results based on our 3Mb FBNN on CORE50 exhibit at par and better performance than conventional real-valued larger NN models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
