Auto-Prox：通过自动代理发现实现无需训练的视觉Transformer架构搜索
Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery
AI总结：
针对现有ViT零成本代理需手动设计、泛化性差的问题，提出Auto-Prox自动代理发现框架，构建ViT-Bench-101基准，通过演化搜索与精英保留策略发现通用零成本代理，实现无需训练的ViT架构搜索并取得SOTA结果。
摘要：
视觉Transformer（ViT）在计算机视觉任务中取得的巨大成功很大程度上归功于架构设计，这凸显了高效架构搜索对于自动设计更优ViT的必要性。由于基于训练的架构搜索方法计算量巨大，人们对使用零成本代理为ViT打分的无需训练方法日益感兴趣。然而，现有的无需训练方法需要专业知识来手动设计特定的零成本代理，此外这些零成本代理在跨不同领域泛化方面存在局限性。本文提出Auto-Prox这一自动代理发现框架来解决该问题。首先，我们构建了ViT-Bench-101，其中包含不同的ViT候选架构及其在多个数据集上的实际性能，借助ViT-Bench-101，我们可以基于分数与准确率的相关性来评估零成本代理。随后，我们用计算图表示零成本代理，并结合ViT统计量与基本操作构建零成本代理搜索空间。为发现通用的零成本代理，我们提出一种联合相关性指标，用于对不同的零成本代理候选进行演化和变异，同时引入精英保留策略提升搜索效率，以在利用与探索之间实现更好的权衡。基于发现的零成本代理，我们以无需训练的方式开展ViT架构搜索。大量实验表明，我们的方法在不同数据集上泛化性良好，在排序相关性和最终准确率方面均达到了当前最优水平。代码可在https://github.com/lilujunai/Auto-Prox-AAAI24获取。
英文摘要：
The substantial success of Vision Transformer (ViT) in computer vision tasks is largely attributed to the architecture design. This underscores the necessity of efficient architecture search for designing better ViTs automatically. As training-based architecture search methods are computationally intensive, there is a growing interest in training-free methods that use zero-cost proxies to score ViTs. However, existing training-free approaches require expert knowledge to manually design specific zero-cost proxies. Moreover, these zero-cost proxies exhibit limitations to generalize across diverse domains. In this paper, we introduce Auto-Prox, an automatic proxy discovery framework, to address the problem. First, we build the ViT-Bench-101, which involves different ViT candidates and their actual performance on multiple datasets. Utilizing ViT-Bench-101, we can evaluate zero-cost proxies based on their score-accuracy correlation. Then, we represent zero-cost proxies with computation graphs and organize the zero-cost proxy search space with ViT statistics and primitive operations. To discover generic zero-cost proxies, we propose a joint correlation metric to evolve and mutate different zero-cost proxy candidates. We introduce an elitism-preserve strategy for search efficiency to achieve a better trade-off between exploitation and exploration. Based on the discovered zero-cost proxy, we conduct a ViT architecture search in a training-free manner. Extensive experiments demonstrate that our method generalizes well to different datasets and achieves state-of-the-art results both in ranking correlation and final accuracy. Codes can be found at https://github.com/lilujunai/Auto-Prox-AAAI24.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
