面向内存高效测试时适应的曲率感知零阶优化
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
- Shanghai Jiao Tong University（上海交通大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文针对零阶测试时适应方法梯度估计方差高的问题，提出CAZO方法，利用损失海森结构优化扰动采样，在降内存开销的同时实现了最优跨域适应性能。
AI中文摘要：
测试时适应（TTA）旨在通过适应无标签测试数据提升预训练模型的跨域性能。现有多数TTA方法依赖反向传播（BP）微调，而零阶（ZO）等无BP方法更适用于实际设备端场景，其仅依赖前向计算，可大幅降低设备端部署的复杂度与内存开销。但ZO方法在梯度估计上的方差远高于一阶方法。为解决该问题，本文提出改进的ZO方法以显著提升ZO优化类TTA的性能：首先，观察到适应过程中损失存在持续的低秩海森结构；基于此，提出损失地形曲率感知零阶（CAZO）方法，该方法利用海森对角项的滑动平均估计构建协方差矩阵，用于各向异性扰动采样。CAZO通过冻结预训练权重，仅优化极小的适配器参数，基于仅前向传播的梯度估计完成，与基于BP的方法相比可大幅降低内存开销。大量实验表明，CAZO显著优于现有TTA方法，在实现最优性能的同时保持了准确率与内存效率的极佳平衡，代码可在该httpsURL获取。
英文摘要：
Test-time adaptation (TTA) aims to enhance the cross-domain performance of pre-trained models by adapting to unlabeled test data. While most existing TTA methods rely on backpropagation (BP) for finetuning, BP-free methods such as zeroth-order (ZO) methods are more desired in practical on-device scenarios. ZO methods rely only on forward computation, which can largely reduce the complexity and memory overhead of on-device deployment. However, ZO methods suffer from much higher variance compared with first-order methods in estimating the gradient. To address this, we propose an improved ZO method to substantially boost the performance of ZO optimization based TTA. First, we provide an observation to reveal the persistent low-rank Hessian structure of the loss during the adaptation process. Based on this insight, we then propose a loss-landscape curvature-aware zeroth-order (CAZO) method, which leverages a sliding-average estimation of the diagonal Hessian to construct a covariance matrix for anisotropic perturbation sampling. CAZO operates by freezing pretrained weights and optimizing minimal adapter parameters via forward-only passes based gradient estimation, which can substantially reduce the memory overhead compared to BP-based methods. Extensive experiments demonstrate that CAZO significantly outperforms existing TTA methods, achieving state-of-the-art performance while maintaining an excellent balance between accuracy and memory efficiency. Code is available at https://github.com/Hollyming/CAZO.
