Q3R：二次加权秩正则化器用于有效的低秩训练
Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
AI总结：
Q3R通过二次加权秩正则化器实现有效的低秩训练，能够在保持预测性能的同时，将权重矩阵训练到指定的低秩目标。
摘要：
基于低秩优化的参数高效训练已成为微调大型深度学习模型的 highly 成功工具。然而，这些方法在低秩预训练中常常失效，其中同时保持低秩权重结构和优化任务目标仍具挑战性。我们提出了二次加权秩正则化器（Q3R），这是一种受迭代加权最小二乘（IRLS）框架启发的新型低秩诱导训练策略。Q3R基于一个二次正则化项，该项主导了平滑对数行列式秩替代项。与其他低秩训练技术不同，Q3R可以在保持预测性能与密集模型相当的情况下，将权重矩阵训练到指定的低目标秩，且计算开销小且与现有架构完全兼容。例如，我们展示了一个加权Q3R的ViT-Tiny实验，其中将模型截断到其参数的60%和80%时，在CIFAR-10上仅导致绝对准确率分别下降1.3%和4%。我们确认了Q3R在视觉和语言任务中的有效性，包括低秩微调。
英文摘要：
Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneously maintaining low-rank weight structure and optimizing the task objective remains challenging. We propose the $\textit{Quadratic Reweighted Rank Regularizer}$ ($\texttt{Q3R}$), which leads to a novel low-rank-inducing training strategy inspired by the Iteratively Reweighted Least Squares (IRLS) framework. $\texttt{Q3R}$ is based on a quadratic regularizer term that majorizes a smoothed log-determinant rank surrogate. Unlike other low-rank training techniques, $\texttt{Q3R}$ can train weight matrices to prescribed low target ranks while achieving predictive performance comparable to dense models, with small computational overhead and full compatibility with existing architectures. For example, we demonstrate a $\texttt{Q3R}$-regularized ViT-Tiny experiment where truncating the model to $60\%$ and $80\%$ of its parameters results in only minor absolute accuracy drops of $1.3\%$ and $4\%$, respectively, on CIFAR-10. We confirm the efficacy of $\texttt{Q3R}$ on Transformers across both vision and language tasks, including low-rank fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
