arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10449cs.LGcs.AI
收藏

随机投影对影响函数的统一理论

A Unified Theory of Random Projection for Influence Functions

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Pingbang Hu, Yuzheng Hu, Jiaqi W. Ma, Han Zhao

AI总结:

本文提出了一种统一理论,探讨随机投影在保持影响函数时的条件,并提供了选择草图大小的指导。

摘要:

影响函数及相关数据归因分数的形式为$g^{\top}F^{-1}g^{\prime}$,其中$F\succeq 0$是一个曲率算子。在现代过参数化模型中,形成或求逆$F\in\mathbb{R}^{d\times d}$是不可行的,这推动了通过随机投影和草图$P \in \mathbb{R}^{m\times d}$进行可扩展的影响计算。这种做法通常通过约翰逊-林德斯特劳斯(JL)引理来证明,该引理确保了固定数据集的欧几里得几何的近似保持。然而,JL并未解决草图在求逆下的行为。此外,目前尚无现有理论能够解释草图与其他广泛使用的技术,如岭正则化和结构化曲率近似如何相互作用。我们开发了一个统一的理论,以确定投影是否能保证影响函数的保持。当$g,g^{\prime}\in\text{range}(F)$时,我们证明:1)无正则化投影:精确保持当且仅当$P$在$\text{range}(F)$上是单射的,这需要$m\geq \text{rank}(F)$;2)正则化投影:岭正则化从根本上改变了草图的障碍,近似保证由$F$在正则化尺度上的有效维度决定;3)因子化影响:对于克罗内克因子化曲率$F=A\otimes E$,保证仍然适用于解耦的草图$P=P_A\otimes P_E$,尽管此类草图表现出行相关性,这违反了i.i.d.假设。除了这个范围受限的设置外,我们分析了超出范围的测试梯度,并量化了当测试梯度具有$\ker(F)$中的成分时出现的泄漏项。这为影响查询在一般测试点上的保证提供了支持。总体而言,这项工作开发了一种新的理论,该理论确定了投影是否能保证影响函数的保持,并为实际选择草图大小提供了指导。

英文摘要:

Influence functions and related data attribution scores take the form of $g^{\top}F^{-1}g^{\prime}$, where $F\succeq 0$ is a curvature operator. In modern overparametrized models, forming or inverting $F\in\mathbb{R}^{d\times d}$ is prohibitive, motivating scalable influence computation via random projection with a sketch $P \in \mathbb{R}^{m\times d}$. This practice is commonly justified via the Johnson--Lindenstrauss (JL) lemma, which ensures approximate preservation of Euclidean geometry for a fixed dataset. However, JL does not address how sketching behaves under inversion. Furthermore, there is no existing theory that explains how sketching interacts with other widely-used techniques, such as ridge regularization and structured curvature approximations. We develop a unified theory characterizing when projection provably preserves influence functions. When $g,g^{\prime}\in\text{range}(F)$, we show that: 1) Unregularized projection: exact preservation holds iff $P$ is injective on $\text{range}(F)$, which necessitates $m\geq \text{rank}(F)$; 2) Regularized projection: ridge regularization fundamentally alters the sketching barrier, with approximation guarantees governed by the effective dimension of $F$ at the regularization scale; 3) Factorized influence: for Kronecker-factored curvatures $F=A\otimes E$, the guarantees continue to hold for decoupled sketches $P=P_A\otimes P_E$, even though such sketches exhibit row correlations that violate i.i.d. assumptions. Beyond this range-restricted setting, we analyze out-of-range test gradients and quantify a leakage term that arises when test gradients have components in $\ker(F)$. This yields guarantees for influence queries on general test points. Overall, this work develops a novel theory that characterizes when projection provably preserves influence and provides principled guidance for choosing the sketch size in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑