How can we attribute the behaviors of machine learning models to their training data? While the classic influence function sheds light on the impact of individual samples, it often fails to capture the more complex and pronounced collective influence of a set of samples. To tackle this challenge, we study the Most Influential Subset Selection (MISS) problem, which aims to identify a subset of training samples with the greatest collective influence. We conduct a comprehensive analysis of the prevailing approaches in MISS, elucidating their strengths and weaknesses. Our findings reveal that influence-based greedy heuristics, a dominant class of algorithms in MISS, can provably fail even in linear regression. We delineate the failure modes, including the errors of influence function and the non-additive structure of the collective influence. Conversely, we demonstrate that an adaptive version of these heuristics which applies them iteratively, can effectively capture the interactions among samples and thus partially address the issues. Experiments on real-world datasets corroborate these theoretical findings and further demonstrate that the merit of adaptivity can extend to more complex scenarios such as classification tasks and non-linear neural networks. We conclude our analysis by emphasizing the inherent trade-off between performance and computational efficiency, questioning the use of additive metrics such as the Linear Datamodeling Score, and offering a range of discussions.
Selection of training populations (and other subset selection problems) with an accelerated genetic algorithm (STPGA: An R-package for selection of training populations with a genetic algorithm)
Semi-Supervised Data Programming with Subset Selection
IIT Bombay(印度理工学院孟买分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校)
数据编程的范式,它使用弱监督的形式的规则/标签功能,和半监督学习,它增加了大量的未标记的数据集少量的标记数据,在几个文本分类的情况下,显示出很大的希望。在这项工作中,我们认为,通过不使用任何标记的数据,基于数据编程的方法可以产生次优性能,特别是当标签函数是嘈杂的。这项工作的第一个贡献是引入一个框架,\model,这是一个半监督的数据编程范式,学习\emph{joint model},有效地使用规则/标签功能以及半监督损失函数的特征空间。接下来,我们还研究了\modelss,它在联合半监督数据编程目标和\emph{selects}的基础上进行子集选择,这是一组可以用作\model标记集的示例。\modelss的目标是确保标记的数据可以通过\emph{complement}标记功能,从而受益于数据编程以及适当选择的人类标记数据。我们证明,通过有效地结合半监督,数据编程和子集选择范式,我们显着优于目前的最先进的七个公开可用的数据集。\footnote{The source code is available at \url{https://github.com/ayushbits/Semi-Supervised-LFs-Subset-Selection}}