超越旋转：用于表达性量化正交微调的AuroOFT
Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning
浏览论文内容
中文总结 AI 辅助
该研究针对量化正交微调（qoft）的线性正交变换局限，提出AuroOFT方法，在保留qoft稳定分支的基础上附加零起点门控低秩非线性残差，在Qwen2.5设置上提升性能并减少可训练参数。
中文摘要 AI 辅助
量化正交微调（qoft）通过在冻结的量化权重前学习结构化激活旋转，实现了低比特语言模型的参数高效适配。然而，其针对特定任务的更新仍局限于线性正交变换，限制了依赖输入的非线性校正。我们提出AuroOFT，它将qoft作为稳定的量化兼容分支保留，同时为每个适配的线性层附加一个零起点门控低秩非线性残差。AuroOFT将激活映射到RMS归一化的紧凑潜在空间，并使用具有有界或token依赖门控的自适应非线性基。零初始化的上投影使AuroOFT在初始化时功能与qoft完全相同，而正交性成为分支级稳定性属性，而非组合非线性层的属性。在匹配的数据、优化、解码和解析协议下，AuroOFT在1.5B/3B Qwen2.5设置上的Macro-6指标较匹配的qoft提升了1.30%-2.70%，较QLoRA超出6.52%-10.62%，且在代表性规模上相比QLoRA节省了32.3%-44.7%的可训练参数。小型考试式多项选择数学集仅作为协议敏感性诊断工具。我们的代码可在匿名仓库获取：this https URL。
英文摘要
Quantized orthogonal fine-tuning (qoft) enables parameter-efficient adaptation of low-bit language models by learning structured activation rotations before frozen quantized weights. However, its task-specific updates remain constrained to linear orthogonal transformations, limiting input-dependent nonlinear corrections. We introduce AuroOFT, which keeps qoft as a stable quantization-compatible branch while attaching a zero-start gated low-rank nonlinear residual to each adapted linear layer. AuroOFT maps activations into an RMS-normalized compact latent space and uses adaptive nonlinear bases with bounded or token-dependent gating. The zero-initialized up projection makes AuroOFT functionally identical to qoft at initialization, while orthogonality remains a branch-level stability property rather than a property of the combined nonlinear layer. Under matched data, optimization, decoding, and parser protocols, AuroOFT improves Macro-6 over matched qoft by 1.30-2.70% on the 1.5B/3B Qwen2.5 settings, exceeds QLoRA by 6.52-10.62%, and saves 32.3-44.7% trainable parameters relative to QLoRA in representative scales. The small exam-style multiple-choice math set is treated only as a protocol-sensitivity diagnostic. Our code is available at the anonymous repository: https://anonymous.4open.science/r/AuroOFT-F3FD.
