arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.14740cs.LGstat.ML
收藏

通过更优的逆 Hessian-向量积实现更好的训练数据归因

Better Training Data Attribution via Better Inverse Hessian-Vector Products

University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(人工智能矢量研究所);Tübingen AI Center, University of Tübingen(图宾根大学人工智能中心);Schwartz Reisman Institute for Technology and Society(技术与社会施瓦茨-雷曼研究所)

Andrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae, Sheila A. McIlraith, Roger Grosse

AI总结:

本文提出 ASTRA 算法,通过 EKFAC 预条件器改进 Neumann 级数迭代,以更少迭代获得更精确的逆 Hessian-向量积近似,从而显著提升训练数据归因性能。

摘要:

训练数据归因(TDA)旨在揭示哪些训练数据对已学习模型的行为负责。基于梯度的方法(如影响函数和展开微分)都涉及一种类似于逆 Hessian-向量积(iHVP)的计算,而该计算难以高效近似。我们提出了一种算法(ASTRA),该算法在 Neumann 级数迭代上使用 EKFAC 预条件器,从而为 TDA 获得精确的 iHVP 近似。ASTRA 易于调参,所需迭代次数少于 Neumann 级数迭代,并且比基于 EKFAC 的近似更准确。通过使用 ASTRA,我们表明提高 iHVP 近似的精度可以显著提升 TDA 性能。

英文摘要:

Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑