arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.18153cs.LGstat.ML
收藏

最具影响力子集选择:挑战、前景与未来

Most Influential Subset Selection: Challenges, Promises, and Beyond

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Yuzheng Hu, Pingbang Hu, Han Zhao, Jiaqi W. Ma

AI总结:

本文研究最具影响力子集选择(MISS)问题,分析基于影响的贪婪方法的失败机制,并证明自适应迭代版本可捕捉样本交互,在真实数据及复杂模型中提升效果。

摘要:

我们如何将机器学习模型的行为归因于其训练数据?尽管经典影响函数有助于揭示单个样本的影响,但它往往无法捕捉一组样本更复杂、更显著的集体影响。为应对这一挑战,我们研究最具影响力子集选择(Most Influential Subset Selection,MISS)问题,其目标是识别具有最大集体影响力的训练样本子集。我们全面分析了MISS中的主流方法,阐明其优势与弱点。研究发现,基于影响的贪婪启发式方法作为MISS中的一类主导算法,即使在线性回归中也可被证明会失败。我们描述了其失败模式,包括影响函数的误差以及集体影响的非加性结构。相反,我们证明这些启发式方法的自适应版本(即迭代应用它们)能够有效捕捉样本间的相互作用,从而部分解决这些问题。在真实世界数据集上的实验证实了这些理论发现,并进一步表明自适应性的优势可以扩展到分类任务和非线性神经网络等更复杂场景。最后,我们强调性能与计算效率之间固有的权衡,质疑Linear Datamodeling Score等加性指标的使用,并给出一系列讨论。

英文摘要:

How can we attribute the behaviors of machine learning models to their training data? While the classic influence function sheds light on the impact of individual samples, it often fails to capture the more complex and pronounced collective influence of a set of samples. To tackle this challenge, we study the Most Influential Subset Selection (MISS) problem, which aims to identify a subset of training samples with the greatest collective influence. We conduct a comprehensive analysis of the prevailing approaches in MISS, elucidating their strengths and weaknesses. Our findings reveal that influence-based greedy heuristics, a dominant class of algorithms in MISS, can provably fail even in linear regression. We delineate the failure modes, including the errors of influence function and the non-additive structure of the collective influence. Conversely, we demonstrate that an adaptive version of these heuristics which applies them iteratively, can effectively capture the interactions among samples and thus partially address the issues. Experiments on real-world datasets corroborate these theoretical findings and further demonstrate that the merit of adaptivity can extend to more complex scenarios such as classification tasks and non-linear neural networks. We conclude our analysis by emphasizing the inherent trade-off between performance and computational efficiency, questioning the use of additive metrics such as the Linear Datamodeling Score, and offering a range of discussions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑