免二次遍历归因：以约1%开销实现内联逐样本梯度溯源
Traceprop: Training Data Attribution in a Single Pass
浏览论文内容
中文总结 AI 辅助
本文提出Traceprop，在训练反向传播中内联记录逐样本梯度，以约1%开销实现数据归因，成本远低于事后方法，并支持欧盟AI法案审计。
中文摘要 AI 辅助
实践中使用的数据归因方法（TRAK、LoGRA、EK-FAC）都是事后型的：在训练完成后，它们需要对训练集进行第二次遍历以重新计算每个样本的梯度，并且在集成时需要对每个检查点重复此操作。Traceprop通过在训练反向传播过程中内联记录投影后的逐样本梯度，避免了这种二次遍历。一种克罗内克因子化的草图（Kronecker-factored sketch）能够从单个被跟踪层扩展到所有层，而无需实例化稠密的投影矩阵。在单块NVIDIA L4上对GPT-2和Pythia模型（最大2.8B参数）进行LoRA微调时，在最后一块（last-block）范围内，内联日志记录的开销仅占挂钟时间的0.30%-1.08%，即使在跟踪Pythia-1B的每一层时，开销也保持在1%以下（0.79%）。与最接近的内联能力竞争对手LogIX相比，在相同存储条件下，因子化草图（factored sketch）的成本低2.0-4.1倍，这一差距随着跟踪范围的扩大而增大，并且在所有测试范围内均显著；同时，在匹配存储条件下，其归因质量达到或超过LogIX。内联构建归因就绪存储比一次事后遍历便宜60-242倍，比五次检查点的TRAK集成便宜301-1211倍，且记录的梯度与autograd完全一致。由于每个存储的梯度都带有源文件谱系（source-file lineage），同一过程还能生成符合欧盟《人工智能法案》第26条的审计追踪。
英文摘要
Data attribution tools used in practice, TRAK, LoGRA/LogIX, and EK-FAC, run after training: they recompute per-sample gradients in a separate pass over the training set, and curvature-aware variants need a further pass to estimate covariance. Traceprop records projected per-sample gradients and K-FAC covariance statistics inside the training backward pass itself. On Pythia-1B LoRA fine-tunes on an A100, this adds 2.6% to training time, against 10.7% for LogIX's best inline configuration, random-init (4.1x lower, one-sided Mann-Whitney p = 9e-5, n = 10 per arm); at Pythia-6.9B the numbers are 11.3% and 27.0% (p = 0.004). LogIX's default configuration, PCA-init, is both slower and lower quality than random-init, so we compare against random-init throughout. On a small transformer where LDS is measurable, Traceprop matches LogIX's best configuration: pooled LDS difference +0.0022, 95% CI [-0.0038, +0.0075]. Two things do not work: at Pythia-160M/SST-2, every method is indistinguishable from a low noise ceiling, and on planted-backdoor and mislabel detection, gradient attribution does not beat gradient norm or representation similarity. The contribution here is systems, not a new estimator: LogIX's attribution quality, in one pass instead of two.
