ROME：通过拓扑解耦和梯度累积增强内存高效的NAS的鲁棒性
ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation
AI总结：
针对单路径DARTS因跳跃连接过多导致性能崩溃的问题，提出ROME算法，通过解耦拓扑与操作搜索、引入Gumbel-Top2重参数化和梯度累积优化双层优化，在15个基准上验证了其有效性与鲁棒性。
摘要：
尽管可微架构搜索（DARTS）是一种流行的架构搜索方法，但由于整个超网络驻留在内存中，其高昂的内存开销极大地阻碍了其应用。单路径DARTS应运而生，它在每一步仅选择单路径子模型。这种方式不仅对内存友好，而且计算成本也很低。然而，我们发现单路径DARTS存在一个尚未引起足够重视的关键问题：与DARTS类似，它也会因为导出过多如跳跃连接等无参数操作而遭遇严重的性能崩溃。在本文中，我们提出了一种名为RObustifying Memory-Efficient NAS（ROME）的新算法来解决这一问题。首先，我们将拓扑搜索与操作搜索解耦，使搜索与评估保持一致。随后，我们采用Gumbel-Top2重参数化和梯度累积来增强难以处理的双层优化。我们在15个基准上对ROME进行了广泛验证，证明了其有效性和鲁棒性。
英文摘要：
Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
