内存约束与神经缩放定律下的行列式估计
Determinant Estimation under Memory Constraints and Neural Scaling Laws
AI总结：
针对大规模正定矩阵对数行列式计算中的内存瓶颈，提出基于LDL分解分块计算的分层算法，并利用神经缩放定律从部分数据准确估计NTK对数行列式，实现约10万倍加速。
摘要：
计算或准确估计大型正定矩阵的对数行列式在许多机器学习任务中具有根本的重要性。虽然其三次方的计算复杂度已经令人望而却步，但在现代应用中，即使仅存储矩阵本身也可能造成内存瓶颈。为了解决这一问题，我们推导出一种新颖的分层算法，该算法基于 LDL 分解的分块计算，用于在内存受限环境下进行大规模对数行列式计算。在矩阵高度病态的极端情况下，准确计算完整矩阵本身可能是不可行的。这在考虑大规模核矩阵时尤为相关，包括在大型数据集上训练的神经网络的经验神经正切核（NTK）。在测试误差遵循神经缩放定律的假设下，我们证明了伪行列式的比率满足幂律关系，从而允许我们推导出相应的缩放定律。这使得能够从完整数据集的极小部分准确估计 NTK 对数行列式；在我们的实验中，这带来了约 100,000 倍的加速，并且与竞争性近似方法相比具有更高的准确性。利用这些技术，我们成功估计了极端规模的稠密矩阵的对数行列式，这些矩阵由于其巨大的规模和计算需求 previously 被认为是难以处理且无法触及的。
英文摘要：
Calculating or accurately estimating log-determinants of large positive definite matrices is of fundamental importance in many machine learning tasks. While its cubic computational complexity can already be prohibitive, in modern applications, even storing the matrices themselves can pose a memory bottleneck. To address this, we derive a novel hierarchical algorithm based on block-wise computation of the LDL decomposition for large-scale log-determinant calculation in memory-constrained settings. In extreme cases where matrices are highly ill-conditioned, accurately computing the full matrix itself may be infeasible. This is particularly relevant when considering kernel matrices at scale, including the empirical Neural Tangent Kernel (NTK) of neural networks trained on large datasets. Under the assumption of neural scaling laws in the test error, we show that the ratio of pseudo-determinants satisfies a power-law relationship, allowing us to derive corresponding scaling laws. This enables accurate estimation of NTK log-determinants from a tiny fraction of the full dataset; in our experiments, this results in a $\sim$100,000$\times$ speedup with improved accuracy over competing approximations. Using these techniques, we successfully estimate log-determinants for dense matrices of extreme sizes, which were previously deemed intractable and inaccessible due to their enormous scale and computational demands.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
