成功之手：压缩深度网络可以提高分发外的鲁棒性
A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
摘要：
在野外成功采用深度学习（DL）需要模型：（1）紧凑，（2）准确，（3）对分布变化具有鲁棒性。不幸的是，同时满足这些要求的努力大多没有成功。这就提出了一个重要的问题：无法创建紧凑、准确和健壮的深度神经网络（CARD）是根本吗？为了回答这个问题，我们进行了大规模的流行的模型压缩技术，揭示了几个有趣的模式分析。值得注意的是，与传统的修剪方法（例如，微调和逐步幅度修剪），我们发现，“彩票式”的方法可以令人惊讶地用于产生CARD，包括二进制权重CARD。具体来说，我们能够创建非常紧凑的CARD，与较大的CARD相比，它们具有相似的测试精度和匹配（或更好）的鲁棒性-只需通过修剪和（可选）量化。利用紧凑的CARD，我们开发了一个简单的域自适应测试时间集成方法（CARD甲板），使用门控模块动态选择适当的CARD从CARD甲板上的光谱相似性与测试样本的基础上。所提出的方法建立了CARD的“胜利之手”，建立了CIFAR-10-C精度的新的最先进水平（在RobustBench上）（即，96.8%标准和92.75%鲁棒）和CIFAR-100-C准确率（80.6%标准和71.3%鲁棒），内存使用率优于非压缩基线（预训练CARD和CARD-Deck可在https：//github.com/RobustBench/robustBenches获得）。最后，我们为我们的实证研究结果提供了理论支持。
英文摘要：
Successful adoption of deep learning (DL) in the wild requires models to be: (1) compact, (2) accurate, and (3) robust to distributional shifts. Unfortunately, efforts towards simultaneously meeting these requirements have mostly been unsuccessful. This raises an important question: Is the inability to create Compact, Accurate, and Robust Deep neural networks (CARDs) fundamental? To answer this question, we perform a large-scale analysis of popular model compression techniques which uncovers several intriguing patterns. Notably, in contrast to traditional pruning approaches (e.g., fine tuning and gradual magnitude pruning), we find that "lottery ticket-style" approaches can surprisingly be used to produce CARDs, including binary-weight CARDs. Specifically, we are able to create extremely compact CARDs that, compared to their larger counterparts, have similar test accuracy and matching (or better) robustness -- simply by pruning and (optionally) quantizing. Leveraging the compactness of CARDs, we develop a simple domain-adaptive test-time ensembling approach (CARD-Decks) that uses a gating module to dynamically select appropriate CARDs from the CARD-Deck based on their spectral-similarity with test samples. The proposed approach builds a "winning hand'' of CARDs that establishes a new state-of-the-art (on RobustBench) on CIFAR-10-C accuracies (i.e., 96.8% standard and 92.75% robust) and CIFAR-100-C accuracies (80.6% standard and 71.3% robust) with better memory usage than non-compressed baselines (pretrained CARDs and CARD-Decks available at https://github.com/RobustBench/robustbench). Finally, we provide theoretical support for our empirical findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
