arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.09447cs.AI
收藏

通过元启发式微调改进预训练权重

Improving Pre-Trained Weights Through Meta-Heuristics Fine-Tuning

São Paulo State University(圣保罗州立大学);São Carlos Federal University(圣卡洛斯联邦大学)

Gustavo H. de Rosa, Mateus Roder, João Paulo Papa, Claudio F. G. dos Santos

摘要:

在过去的十年中,机器学习算法得到了广泛的研究,在广泛的应用中取得了前所未有的进步,例如图像分类和重建,对象识别和文本分类。尽管如此,大多数机器学习算法都是通过基于导数的优化器(例如随机梯度下降)进行训练的,这可能会导致局部最优捕获并抑制它们实现适当的性能。一个生物启发的替代传统的优化技术,表示为元启发式,已受到显着的关注,由于其简单性和能力,以避免局部最优监禁。在这项工作中,我们建议使用元启发式技术来微调预训练的权重,探索搜索空间的其他区域,并提高其有效性。实验评估包括两个分类任务(图像和文本),并在四个文献数据集下进行评估。实验结果表明,自然启发式算法的能力,探索邻域的预训练的权重,实现优于其对应的预训练的架构。此外,对不同架构(如多层感知器和递归神经网络)的全面分析试图可视化并提供对学习过程中需要微调的最关键权重的更精确见解。

英文摘要:

Machine Learning algorithms have been extensively researched throughout the last decade, leading to unprecedented advances in a broad range of applications, such as image classification and reconstruction, object recognition, and text categorization. Nonetheless, most Machine Learning algorithms are trained via derivative-based optimizers, such as the Stochastic Gradient Descent, leading to possible local optimum entrapments and inhibiting them from achieving proper performances. A bio-inspired alternative to traditional optimization techniques, denoted as meta-heuristic, has received significant attention due to its simplicity and ability to avoid local optimums imprisonment. In this work, we propose to use meta-heuristic techniques to fine-tune pre-trained weights, exploring additional regions of the search space, and improving their effectiveness. The experimental evaluation comprises two classification tasks (image and text) and is assessed under four literature datasets. Experimental results show nature-inspired algorithms' capacity in exploring the neighborhood of pre-trained weights, achieving superior results than their counterpart pre-trained architectures. Additionally, a thorough analysis of distinct architectures, such as Multi-Layer Perceptron and Recurrent Neural Networks, attempts to visualize and provide more precise insights into the most critical weights to be fine-tuned in the learning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑