arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.14838cs.NEcs.AIcs.CLcs.LG
收藏

EvoPrompting:用于代码级神经架构搜索的语言模型

EvoPrompting: Language Models for Code-Level Neural Architecture Search

New York University(纽约大学);Google DeepMind(谷歌DeepMind);OpenAI(开放人工智能);Jane Street(简街)

Angelica Chen, David M. Dohan, David R. So

AI总结:

提出EvoPrompting方法,结合进化提示工程与软提示微调,利用语言模型作为变异和交叉算子进行神经架构搜索,在MNIST-1D和CLRS基准上生成优于人类设计和现有最先进模型的架构。

摘要:

鉴于语言模型(LMs)在代码生成方面近期取得的令人瞩目的成就,我们探索将语言模型用作进化神经架构搜索(NAS)算法中的自适应变异和交叉算子。尽管仅通过提示(prompting)让语言模型成功完成NAS任务仍然过于困难,但我们发现,将进化提示工程(evolutionary prompt engineering)与软提示微调(soft prompt-tuning)相结合的方法(我们称之为EvoPrompting)能够持续找到多样且高性能的模型。我们首先在计算高效的MNIST-1D数据集上证明了EvoPrompting的有效性,在该数据集上,EvoPrompting生成的卷积架构变体在准确性和模型大小方面均优于由人类专家设计的架构和朴素少样本提示(few-shot prompting)生成的架构。随后,我们将我们的方法应用于在CLRS算法推理基准上搜索图神经网络,在该基准上,EvoPrompting能够设计出新颖的架构,在30个算法推理任务中的21个上优于当前最先进的模型,同时保持相似的模型大小。EvoPrompting成功地在各种机器学习任务中设计出准确且高效的神经网络架构,同时其通用性足以轻松适应神经网络设计之外的其他任务。

英文摘要:

Given the recent impressive accomplishments of language models (LMs) for code generation, we explore the use of LMs as adaptive mutation and crossover operators for an evolutionary neural architecture search (NAS) algorithm. While NAS still proves too difficult a task for LMs to succeed at solely through prompting, we find that the combination of evolutionary prompt engineering with soft prompt-tuning, a method we term EvoPrompting, consistently finds diverse and high performing models. We first demonstrate that EvoPrompting is effective on the computationally efficient MNIST-1D dataset, where EvoPrompting produces convolutional architecture variants that outperform both those designed by human experts and naive few-shot prompting in terms of accuracy and model size. We then apply our method to searching for graph neural networks on the CLRS Algorithmic Reasoning Benchmark, where EvoPrompting is able to design novel architectures that outperform current state-of-the-art models on 21 out of 30 algorithmic reasoning tasks while maintaining similar model size. EvoPrompting is successful at designing accurate and efficient neural network architectures across a variety of machine learning tasks, while also being general enough for easy adaptation to other tasks beyond neural network design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑