arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.15781cs.LGcs.AIcs.CL
收藏

FastIF:用于高效模型解释和建模的可扩展影响函数

FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

Salesforce Research(Salesforce研究院);Carnegie Mellon University(卡内基梅隆大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校)

Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, Caiming Xiong

摘要:

影响函数近似于测试预测的训练数据点的“影响”,并且具有广泛的应用。尽管很流行,但它们的计算成本并不随模型和训练数据的大小而变化。我们提出了FastIF,一组简单的修改,以影响功能,显着提高其运行时间。我们使用k-近邻(kNN)缩小搜索空间的一个子集的良好的候选数据点,确定的配置,最好的平衡的速度和质量的权衡估计逆海森矢量积,并引入一个快速的并行变量。我们提出的方法实现了约80倍的加速比,同时与原始影响值高度相关。随着快速影响函数的可用性,我们证明了它们在四个应用程序中的有用性。首先,我们研究是否有影响力的数据点可以“解释”测试时间的行为使用的框架的可模拟性。其次,我们可视化的影响之间的相互作用的训练和测试数据-点。第三,我们证明了我们可以通过对某些有影响力的数据点进行额外的微调来纠正模型错误,将训练后的MultiNLI模型在HANS数据集上的准确性提高了2.5%。最后,我们使用类似的设置进行实验,但对训练过程中未看到的数据点进行微调,在HANS和ANLI数据集上分别将模型准确度提高了2.8%和1.7%。总的来说,我们的快速影响函数可以有效地应用于大型模型和数据集,我们的实验证明了影响函数在模型解释和纠正模型错误方面的潜力。代码可在https://github.com/salesforce/fast-influence-functions上获得

英文摘要:

Influence functions approximate the "influences" of training data-points for test predictions and have a wide variety of applications. Despite the popularity, their computational cost does not scale well with model and training data size. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. We use k-Nearest Neighbors (kNN) to narrow the search space down to a subset of good candidate data points, identify the configurations that best balance the speed-quality trade-off in estimating the inverse Hessian-vector product, and introduce a fast parallel variant. Our proposed method achieves about 80X speedup while being highly correlated with the original influence values. With the availability of the fast influence functions, we demonstrate their usefulness in four applications. First, we examine whether influential data-points can "explain" test time behavior using the framework of simulatability. Second, we visualize the influence interactions between training and test data-points. Third, we show that we can correct model errors by additional fine-tuning on certain influential data-points, improving the accuracy of a trained MultiNLI model by 2.5% on the HANS dataset. Finally, we experiment with a similar setup but fine-tuning on datapoints not seen during training, improving the model accuracy by 2.8% and 1.7% on HANS and ANLI datasets respectively. Overall, our fast influence functions can be efficiently applied to large models and datasets, and our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors. Code is available at https://github.com/salesforce/fast-influence-functions

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑