AI 中文总结
该研究发现行为有效的LoRA更新稀疏且结构化，通过Learned-Basis LoRA揭示其结构，证实更新几何是因果状态变量，且关键组件集中在q_proj等模块，在数学推理数据集上验证了top-k更新的最优性。
AI 中文摘要
低秩自适应（LoRA）确定了更新的秩，但未识别训练后的更新中哪些部分承载着行为。我们直接研究这一问题，发现行为上有效的LoRA更新是稀疏、结构化的，且比原始低秩参数化所显示的更集中。我们使用Learned-Basis LoRA（一种学习基的延续方案）来揭示这种结构：该方案先预热一个无约束适配器，将其学习到的更新列转换为模块级正交基，冻结该基后在受限参数化内继续训练。在14次从无约束形式到受限形式的精确切换中，转换步骤的保留准确率保持不变，重构的更新矩阵的相对Frobenius误差最多为0.25%。相同状态的延续实验显示，同一训练检查点在不同的更新子空间下会产生不同的发展，这确立了更新几何作为因果状态变量的作用。无重训练投影测试表明，有用的更新信号保留在学习到的更新空间内，而在随机或冻结激活的PCA对照组中基本消失。这种集中模式在局部和全局尺度上都很强。在GSM8K、MathQA和AQuA数据集上，模块级的top-k延续在我们测试的所有12个种子级案例中，均在k取{2,4}时达到最优。更严格的全局排名测试显示，学习到的top-16和top-32子集优于匹配的随机子集，尤其在GSM8K/Qwen和MathQA/Qwen上表现突出。单向消融实验进一步揭示了一组稀疏的late q_proj、o_proj和down_proj组件，它们具有巨大的行为影响。
英文摘要
Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior. We study that question directly and show that behaviorally effective LoRA writes are sparse, structured, and far more concentrated than the raw low-rank parameterization suggests. We use Learned-Basis LoRA, a learned-basis continuation recipe, to expose that structure. The recipe warms up an unconstrained adapter, converts its learned write columns into a module-wise orthonormal basis, freezes that basis, and continues training inside the constrained parameterization. Across 14 exact switches from unconstrained to constrained form, held-out accuracy is unchanged at the conversion step and reconstructed write matrices differ by at most 0.25% relative Frobenius error. Same-state continuation then shows that the same trained checkpoint develops differently under different write subspaces, establishing write geometry as a causal state variable. A no-retraining projection test shows that useful write signal stays inside the learned write space and largely disappears from random or frozen-activation PCA controls. The concentration pattern is strong at both local and global scales. Across GSM8K, MathQA, and AQuA, per-module top-k continuation reaches its optimum at k in {2, 4} in all twelve seed-level cases we test. A stricter global ranking test shows that learned top-16 and top-32 subsets outperform matched random subsets, especially on GSM8K/Qwen and MathQA/Qwen. Single-direction ablations further reveal a sparse set of late q_proj, o_proj, and down_proj components with outsized behavioral impact.
