arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.12237cs.LGcs.AI
收藏

面向资源受限物联网环境的高效基于Transformer的超参数优化

Efficient Transformer-based Hyper-parameter Optimization for Resource-constrained IoT Environments

University of Western Ontario(西安大略大学)

Ibrahim Shaer, Soodeh Nikan, Abdallah Shami

AI总结:

本文提出TRL-HPO,将Transformer架构与演员-评论家强化学习结合,利用多头注意力实现并行化和渐进式层生成,在MNIST上比最先进方法分类性能提升6.8%,并指出堆叠全连接层是性能下降主因。

摘要:

超参数优化(HPO)过程对于找到性能最佳的卷积神经网络(CNN)至关重要。HPO的自动化过程以其庞大的计算足迹和缺乏透明性为特征;这两者在资源受限的物联网(IoT)环境中都是重要因素。在本文中,我们通过提出一种新颖的方法来解决这些问题,该方法结合了Transformer架构和演员-评论家强化学习(RL)模型,即TRL-HPO,并配备多头注意力,从而实现并行化和层的渐进式生成。这些假设通过在MNIST数据集上评估TRL-HPO,并将其与从头构建CNN模型的最先进方法进行比较,得到了实证支持。结果表明,在相同时间范围内,TRL-HPO的分类结果比这些方法高出6.8%,证明了TRL-HPO在HPO过程中的效率。结果分析指出,导致性能下降的主要原因是堆叠全连接层。本文为在资源受限环境中改进基于RL的HPO过程确定了新的途径。

英文摘要:

The hyper-parameter optimization (HPO) process is imperative for finding the best-performing Convolutional Neural Networks (CNNs). The automation process of HPO is characterized by its sizable computational footprint and its lack of transparency; both important factors in a resource-constrained Internet of Things (IoT) environment. In this paper, we address these problems by proposing a novel approach that combines transformer architecture and actor-critic Reinforcement Learning (RL) model, TRL-HPO, equipped with multi-headed attention that enables parallelization and progressive generation of layers. These assumptions are founded empirically by evaluating TRL-HPO on the MNIST dataset and comparing it with state-of-the-art approaches that build CNN models from scratch. The results show that TRL-HPO outperforms the classification results of these approaches by 6.8% within the same time frame, demonstrating the efficiency of TRL-HPO for the HPO process. The analysis of the results identifies the main culprit for performance degradation attributed to stacking fully connected layers. This paper identifies new avenues for improving RL-based HPO processes in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑