放大影响函数
Scaling Up Influence Functions
摘要:
我们解决了有效的计算影响函数的跟踪预测回训练数据。提出并分析了一种基于Arnoldi迭代的Hessian逆计算加速方法。通过这种改进,据我们所知,我们实现了影响函数的首次成功实现,该影响函数可扩展到具有数亿参数的全尺寸(语言和视觉)Transformer模型。我们评估我们的方法对图像分类和序列到序列的任务与数以千万计的训练样本。我们的代码将在https://github.com/google-research/jax-influence上提供。
英文摘要:
We address efficient calculation of influence functions for tracking predictions back to the training data. We propose and analyze a new approach to speeding up the inverse Hessian calculation based on Arnoldi iteration. With this improvement, we achieve, to the best of our knowledge, the first successful implementation of influence functions that scales to full-size (language and vision) Transformer models with several hundreds of millions of parameters. We evaluate our approach on image classification and sequence-to-sequence tasks with tens to a hundred of millions of training examples. Our code will be available at https://github.com/google-research/jax-influence.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。