学习记忆：为紧凑型循环神经网络提炼记忆保持能力
Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks
浏览论文内容
中文总结 AI 辅助
针对现有知识蒸馏忽视时间序列模型记忆特性，提出记忆差异知识蒸馏（MemKD）框架，通过捕捉师生模型子序列记忆差异，实现紧凑循环神经网络的高性能压缩。
中文摘要 AI 辅助
深度学习模型，特别是循环神经网络及其变体（如长短期记忆网络），已显著推进了时间序列分析。这些模型能够捕捉时间序列中复杂的顺序模式，从而实现实时评估。然而，它们的高计算复杂性和大模型规模给在资源受限环境（如可穿戴设备和边缘计算平台）中的部署带来了挑战。知识蒸馏（KD）提供了一种解决方案，通过将知识从大型复杂模型（教师）转移到更小、更高效的模型（学生），从而在降低计算需求的同时保持高性能。当前的KD方法最初是为计算机视觉任务设计的，忽视了时间序列模型独特的时序依赖和记忆保持特性。为弥补这一差距，我们提出了一种新颖的KD框架，称为记忆差异知识蒸馏（MemKD）。MemKD利用专门的损失函数来捕捉教师和学生模型在时间序列数据子序列间的记忆保持差异，确保学生模型有效模仿教师的行为。该方法促进了适用于实时时间序列分析任务的紧凑、高性能循环神经网络的开发。我们提供了额外的实验、深入的理论分析以及对所提框架在扩展时间序列基准上的见解。我们的实验表明，MemKD显著优于最先进的KD方法。此外，我们证明它能在广泛的压缩水平上匹配教师模型的性能，在参数数量和内存使用上实现显著减少，而准确性损失不大。
英文摘要
Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time series analysis. These models capture complex, sequential patterns in time series, enabling real-time assessments. However, their high computational complexity and large model sizes pose challenges for deployment in resource-constrained environments, such as wearable devices and edge computing platforms. Knowledge Distillation (KD) offers a solution by transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student), thereby retaining high performance while reducing computational demands. Current KD methods, originally designed for computer vision tasks, neglect the unique temporal dependencies and memory retention characteristics of time series models. To bridge this gap, we propose a novel KD framework termed Memory-Discrepancy Knowledge Distillation (MemKD). MemKD leverages a specialized loss function to capture memory retention discrepancies between the teacher and student models across subsequences within time series data, ensuring that the student model effectively mimics the teacher's behaviour. This approach facilitates the development of compact, high-performing recurrent neural networks suitable for real-time, time series analysis tasks. We provide additional experiments, in-depth theoretical analysis, and insights into the proposed framework across extended time series benchmarks. Our experiments demonstrate that MemKD significantly outperforms state-of-the-art KD methods. Additionally, we demonstrate that it can match the teacher model's performance across a wide range of compression levels, achieving notable reductions in parameter count and memory usage without a significant loss in accuracy.
发表机构
- University of Melbourne（墨尔本大学）
机构由 AI 辅助整理，请以论文原文为准。
