发表机构
University of California, Los Angeles（加州大学洛杉矶分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出信号-噪声分解正则化，在CIFAR-100和BloodMNIST上提升性能，通过将噪声隔离到可移除子空间，显著改善分布外图像失真下的模型准确率。
AI 中文摘要
近期的理论工作识别了表征几何的基本性质，这些性质塑造了深度神经网络的推理能力。其中包括信号-噪声分解（SNF），即将信号与噪声分离的能力，以及信号-信号分解（SSF），即将任务特定信号与任务无关信号分离的能力。在此，我们构建了在训练过程中强化这两种性质的正则化器。我们在CIFAR-100分类任务上将使用这些正则化器训练的网络与L2正则化基线网络进行比较，以理解我们的正则化器如何塑造表征几何并影响在著名计算机视觉基线任务上的性能。通过正则化增强SNF改善了模型性能，但增强SSF并未带来改善。受生物医学应用的启发，我们研究了正则化器对BloodMNIST数据集（经MedMNIST-C五种严重程度的损坏处理）性能的影响，发现使用SNF正则化器获得了更大的性能提升。为了理解SNF正则化产生性能提升的机制，我们分析了不同正则化机制下的干扰子空间，发现SNF正则化模型将噪声表示在独立于类别相关信号的独特子空间中。由于这种几何结构是显式的，可以在保留数据上估计由损坏主导的方向，并将其从表征中投影出去。这种操作带来了准确率的显著提升。这些结果表明，强制信号-噪声分解的正则化器可以在推理时包含分布外图像失真的计算机视觉任务上产生实质性改进。它们还强调了塑造表征如何影响模型性能：将干扰变量与类别变量隔离比维持类别变量的分解表征更为重要。
英文摘要
Recent theoretical work identified fundamental properties of representation geometry that shape inference ability of deep neural networks. These include signal-noise factorization (SNF), the ability to segregate signal from noise, and signal-signal factorization (SSF), the ability to segregate task-specific and task-irrelevant signals. Here, we built regularizers that reinforce these two properties during training. We compared networks trained with these regularizers to $L_2$-regularized baseline networks on the CIFAR-100 classification task to understand how our regularizers shape representation geometry and impact performance on a well-known computer vision baseline. Enhancing SNF via regularization improved model performance but enhancing SSF did not. Motivated by biomedical applications, we investigated how our regularizers affected performance on the BloodMNIST dataset treated with MedMNIST-C corruptions at five severity levels, and found even larger performance gains using the SNF regularizer. To understand the mechanism by which SNF-regularization produces improved performance, we analyzed the nuisance subspaces across regularization regimes, finding that the SNF-regularized models represent noise in distinct subspaces, separate from class-relevant signal. Because this geometry is explicit, the dominant corruption-induced directions can be estimated on held-out data and projected out of the representations. This manipulation led to a substantial gain in accuracy. These results show that regularizers that enforce signal-noise factorization can produce substantial improvements on computer vision tasks that contain out-of-distribution image distortions at inference time. They also highlight how shaping representations affects model performance: isolating nuisance variables from categorical ones is more important than maintaining factorized representations of categorical variables.
