arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.02038cs.LG
收藏

NAS-OoD:面向分布外泛化的神经结构搜索

NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization

Shanghai Jiao Tong University(上海交通大学)

Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Ye, S. -H. Gary Chan, Zhenguo Li

摘要:

分布外(OoD)泛化的最新进展揭示了深度学习模型对分布偏移的鲁棒性。然而,现有的工作集中在OoD算法,如不变风险最小化,领域泛化,或稳定学习,没有考虑深度模型架构对OoD泛化的影响,这可能导致次优性能。神经架构搜索(NAS)方法基于架构在训练数据上的性能来搜索架构,这可能导致OoD任务的泛化能力差。在这项工作中,我们提出了强大的神经架构搜索OoD泛化(NAS-OoD),它优化了架构方面的性能生成的OoD数据的梯度下降。具体来说,数据生成器学习通过最大化由不同神经架构计算的损失来合成OoD数据,而架构搜索的目标是找到最佳架构参数,使合成OoD数据损失最小化。数据生成器和神经架构以端到端的方式联合优化,并且极大极小训练过程有效地发现了针对不同分布偏移很好地泛化的鲁棒架构。大量的实验结果表明,NAS-OoD在各种OoD泛化基准测试中表现出色,深度模型的参数数量少得多。此外,在实际工业数据集上,与现有方法相比,NAS-OoD方法的出错率降低了70%以上,证明了该方法的实用性.

英文摘要:

Recent advances on Out-of-Distribution (OoD) generalization reveal the robustness of deep learning models against distribution shifts. However, existing works focus on OoD algorithms, such as invariant risk minimization, domain generalization, or stable learning, without considering the influence of deep model architectures on OoD generalization, which may lead to sub-optimal performance. Neural Architecture Search (NAS) methods search for architecture based on its performance on the training data, which may result in poor generalization for OoD tasks. In this work, we propose robust Neural Architecture Search for OoD generalization (NAS-OoD), which optimizes the architecture with respect to its performance on generated OoD data by gradient descent. Specifically, a data generator is learned to synthesize OoD data by maximizing losses computed by different neural architectures, while the goal for architecture search is to find the optimal architecture parameters that minimize the synthetic OoD data losses. The data generator and the neural architecture are jointly optimized in an end-to-end manner, and the minimax training process effectively discovers robust architectures that generalize well for different distribution shifts. Extensive experimental results show that NAS-OoD achieves superior performance on various OoD generalization benchmarks with deep models having a much fewer number of parameters. In addition, on a real industry dataset, the proposed NAS-OoD method reduces the error rate by more than 70% compared with the state-of-the-art method, demonstrating the proposed method's practicality for real applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑