发表机构
HSE University（高等经济大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文研究一类由块对角因子与固定置换构成的低参数正交矩阵，通过黎曼几何推导其流形结构，提出高效优化算法，并应用于矩阵逼近与大语言模型微调。
AI 中文摘要
本文研究由块对角因子与固定置换交错形成的矩阵——一类灵活的结构化矩阵族。该类矩阵因其在表达性与效率之间的平衡优势，近期在深度学习架构中引起关注，然而针对其高效的计算策略仍有待探索。我们通过黎曼几何途径处理此问题，考察该类矩阵在何种条件下具有光滑流形结构。对于实际重要的正交双因子矩阵情形，我们推导了必要的黎曼工具，并提出高效的实现算法。这些算法利用自动微分，支持每个因子内的参数共享，并避免显式构造稠密矩阵。我们在黎曼优化框架下，将其应用于最佳矩阵逼近问题以及大型语言模型的参数高效微调。在双因子情形之外，我们还研究了具有更多块对角因子的分解的几何与矩阵理论性质。
英文摘要
In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.
Comments37 pages, 2 figures
