DiReDi：面向AIoT应用的蒸馏与反向蒸馏
DiReDi: Distillation and Reverse Distillation for AIoT Applications
AI总结：
针对AIoT场景下边缘AI模型定制难、本地微调易引发故障与隐私风险的问题，提出包含知识蒸馏与反向蒸馏的DiReDi框架，通过仅回传提取的知识更新云侧模型再下发更新边缘模型，在保护隐私的同时实现模型适配与冗余知识削减。
摘要：
通常情况下，在各类真实场景中部署不同的边缘AI模型，同时由少量大型模型在云端服务器远程管理这些边缘AI模型，可实现显著的效率提升。然而，为每个用户的特定应用定制边缘AI模型，或是将现有模型扩展到新的应用场景，仍然是一项挑战。用户对边缘AI模型进行不当的本地训练或微调可能导致模型故障，进而可能给制造商带来法律问题。为解决上述问题，本文提出了一种名为“DiReD”的创新框架，该框架包含知识蒸馏与反向蒸馏机制。在初始步骤中，利用预设数据训练边缘AI模型，并通过上层管理云服务器中的云端AI模型执行知识蒸馏（KD）过程。随后，该边缘AI模型被下发到边缘AI设备，仅用于在用户的应用场景中执行推理。当用户需要更新边缘AI模型以更好地适配实际场景时，采用反向蒸馏（RD）过程提取知识：即利用用户的专属数据，从边缘AI模型中提取用户偏好与制造商预设之间的差异。仅将提取到的知识上报给上层管理云服务器以更新云端AI模型，从而通过不使用任何专属数据的方式保护用户隐私。更新后的云端AI模型随后可利用扩展后的知识更新边缘AI模型。仿真结果表明，所提出的“DiReDi”框架使制造商能够通过从用户带有隐私数据的实际场景中学习新知识来更新用户模型。由于重新训练过程侧重用户隐私数据，初始的冗余知识得到了减少。
英文摘要：
Typically, the significant efficiency can be achieved by deploying different edge AI models in various real world scenarios while a few large models manage those edge AI models remotely from cloud servers. However, customizing edge AI models for each user's specific application or extending current models to new application scenarios remains a challenge. Inappropriate local training or fine tuning of edge AI models by users can lead to model malfunction, potentially resulting in legal issues for the manufacturer. To address aforementioned issues, this paper proposes an innovative framework called "DiReD", which involves knowledge DIstillation & REverse DIstillation. In the initial step, an edge AI model is trained with presumed data and a KD process using the cloud AI model in the upper management cloud server. This edge AI model is then dispatched to edge AI devices solely for inference in the user's application scenario. When the user needs to update the edge AI model to better fit the actual scenario, the reverse distillation (RD) process is employed to extract the knowledge: the difference between user preferences and the manufacturer's presumptions from the edge AI model using the user's exclusive data. Only the extracted knowledge is reported back to the upper management cloud server to update the cloud AI model, thus protecting user privacy by not using any exclusive data. The updated cloud AI can then update the edge AI model with the extended knowledge. Simulation results demonstrate that the proposed "DiReDi" framework allows the manufacturer to update the user model by learning new knowledge from the user's actual scenario with private data. The initial redundant knowledge is reduced since the retraining emphasizes user private data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
