基于Stein变分梯度下降在Stiefel流形上的几何感知贝叶斯参数高效微调
Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent
浏览论文内容
中文总结 AI 辅助
本文提出基于Stein变分梯度下降的几何感知贝叶斯低秩适配方法，在Stiefel流形上优化，实现更好的不确定性量化与模型校准，并提升预测准确率。
中文摘要 AI 辅助
针对大型预训练模型的参数高效微调，近期涌现出若干几何感知的低秩适配方法。这些方法旨在充分利用低秩流形的几何结构，通过在优化过程中施加正交性约束，提高子空间利用效率并减少冗余。这些技术的强实证结果促使进一步研究此类基于几何的适配方法的预测是否可能过度自信。本文基于适配器的奇异值分解因子化，构建了一个基于Stein变分梯度下降（SVGD）的框架。在该框架中，低秩矩阵沿Stiefel流形传输以匹配目标分布，同时保留其关键的几何结构。由于这种几何感知的SVGD方法在推理过程中提供多个解，它支持不确定性量化，并在Stiefel流形上产生校准更好的适配器。大量实验表明，我们的方法实现了强模型校准，并在预测准确率上优于在欧几里得空间中制定的SVGD及相关不确定性估计方法。
英文摘要
Several geometry-aware approaches to low-rank adaptation have emerged for parameter-efficient fine-tuning of large pre-trained models. These methods aim to take full advantage of the geometric structure of low-rank manifolds for improving the efficiency in subspace utilization and reducing redundancy by enforcing orthogonality constraints during optimization. The strong empirical results of these techniques have motivated further study into whether predictions from such geometry-based adaptation methods could be overconfident. In this paper, we build on the singular value decomposition factorization of adapters to develop a framework based on Stein variational gradient descent (SVGD). In this formulation, the low-rank matrices are transported along the Stiefel manifold to match the targeted distributions while retaining their crucial geometric structure. Since this geometry-aware SVGD approach provides multiple solutions during inference, it supports uncertainty quantification and produces better-calibrated adapters on the Stiefel manifold. Extensive experiments show that our method delivers strong model calibration and attains higher prediction accuracy than SVGD and related uncertainty estimation methods that are formulated in Euclidean space.
发表机构
- Deakin University（迪肯大学）
- Aalto University（阿尔托大学）
- Monash University（莫纳什大学）
机构由 AI 辅助整理，请以论文原文为准。
