arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00836cs.ITcs.DCcs.LGmath.IT
收藏

重新思考边缘学习中的资源管理:一种预训练与微调联合设计范式

Rethinking Resource Management in Edge Learning: A Joint Pre-training and Fine-tuning Design Paradigm

The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));School of Science and Engineering (SSE)(理工学院);Future Network of Intelligence Institute (FNii)(未来网络智能研究院);Shanghai Jiao Tong University(上海交通大学);Baidu, Inc.(百度公司);Shenzhen Research Institute of Big Data(深圳大数据研究院);Princeton University(普林斯顿大学)

Zhonghao Lyu, Yuchen Li, Guangxu Zhu, Jie Xu, H. Vincent Poor, Shuguang Cui

AI总结:

本文针对两阶段边缘学习系统,提出联合优化预训练与微调阶段通信和计算资源的框架,以最小化收敛误差界并平衡训练精度、时延与能耗。

摘要:

在一些应用中,边缘学习正经历从传统的从零开始学习向一种新的两阶段学习的转变,该两阶段学习将预训练与任务特定的微调整合在一起。本文考虑了一个两阶段边缘学习系统中的通信与计算资源联合管理问题。在该系统中,模型预训练首先在边缘服务器上通过对本地预存的通用数据进行集中式学习来完成,然后基于预训练模型,通过联邦边缘学习在边缘设备上执行任务特定的微调。针对该两阶段学习模型,我们首先分析了其收敛行为(以平均平方梯度范数界为指标),刻画了诸如两个阶段中的学习轮数和批量大小等各种系统参数对收敛速率的影响。基于我们的分析结果,我们随后提出了一种通信与计算资源联合管理设计,以在发射功率、系统总能耗和训练时延的约束下最小化平均平方梯度范数界。决策变量包括预训练和微调两个阶段的学习轮数、批量大小、时钟频率以及发射功率控制。最后,提供了数值结果以评估我们所提设计的有效性。结果表明,所提出的跨预训练和微调阶段的联合资源管理能够很好地平衡训练精度、时延和能耗之间的系统性能折中。该设计还被证明能够有效利用预训练与微调之间固有的折中关系,这种折中源于预存通用数据与实时任务特定数据之间数据分布的差异,从而高效地优化整体系统性能。

英文摘要:

In some applications, edge learning is experiencing a shift in focusing from conventional learning from scratch to new two-stage learning unifying pre-training and task-specific fine-tuning. This paper considers the problem of joint communication and computation resource management in a two-stage edge learning system. In this system, model pre-training is first conducted at an edge server via centralized learning on local pre-stored general data, and then task-specific fine-tuning is performed at edge devices based on the pre-trained model via federated edge learning. For the two-stage learning model, we first analyze the convergence behavior (in terms of the average squared gradient norm bound), which characterizes the impacts of various system parameters such as the number of learning rounds and batch sizes in the two stages on the convergence rate. Based on our analytical results, we then propose a joint communication and computation resource management design to minimize an average squared gradient norm bound, subject to constraints on the transmit power, overall system energy consumption, and training delay. The decision variables include the number of learning rounds, batch sizes, clock frequencies, and transmit power control for both pre-training and fine-tuning stages. Finally, numerical results are provided to evaluate the effectiveness of our proposed design. It is shown that the proposed joint resource management over the pre-training and fine-tuning stages well balances the system performance trade-off among the training accuracy, delay, and energy consumption. The proposed design is also shown to effectively leverage the inherent trade-off between pre-training and fine-tuning, which arises from the differences in data distribution between pre-stored general data versus real-time task-specific data, thus efficiently optimizing overall system performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑