arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.06910cs.LG
收藏

生成流诱导的神经架构搜索:面向小波神经算子中的最优架构发现

Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator

Indian Institute of Technology Delhi(印度德里理工学院);Yardi School of Artificial Intelligence (ScAI)(亚迪人工智能学院)

Hartej Soin, Tapas Tripura, Souvik Chakraborty

AI总结:

本研究提出一种生成流诱导的神经架构搜索算法,通过前馈网络学习策略生成WNO超参数序列,以终态奖励引导搜索,在流体力学问题上提升了原始小波神经算子的性能。

摘要:

我们提出了一种生成流诱导的神经架构搜索算法。该方法利用简单的前馈神经网络学习随机策略,以生成架构超参数序列,使生成的状态与来自终态的奖励成比例。我们在小波神经算子(WNO,wavelet neural operator)上验证了所提搜索算法的有效性,在此场景中我们学习一种策略,为小波积分块生成诸如小波基、激活算子等超参数序列。生成的小波基与激活序列的轨迹被建模为流,策略通过最小化轨迹中每个状态间的流违背、同时最大化终态奖励来学习。在终态阶段,我们同步训练WNO以引导搜索过程,并提出将验证集上WNO损失的负数的指数作为奖励函数。基于网格搜索的神经架构生成算法需要遍历所有组合,而所提框架基于终态的正奖励生成概率最高的序列,由此缩短了探索时间。与需要完成完整回合训练才能获取奖励的强化学习方案不同,所提算法可按顺序生成超参数轨迹。通过4个面向流体力学的问题,我们证明学习到的策略能够采样出性能最优的神经算子架构,从而提升了原始小波神经算子的性能。

英文摘要:

We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑