arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.17867stat.MLcs.AIcs.LG
收藏

多示例学习中的可复现性:算法单元测试的一个案例

Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests

Booz Allen Hamilton(博思艾伦汉密尔顿公司);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Laboratory for Physical Sciences(物理科学实验室)

Edward Raff, James Holt

AI总结:

本文通过提出算法单元测试,发现五个主流深度多示例学习模型均违反标准MIL假设,能够学习反相关实例,并提供了模型无关的假设违反识别方法,以降低实际应用中学习错误模型的风险。

摘要:

多示例学习(Multiple Instance Learning, MIL)是分类问题的一个子领域,具有正负标签和一个由输入组成的“包”,当且仅当包中包含一个正例元素时,该包的标签为正,否则为负。在这种情境下进行训练需要将包级标签与实例级信息关联起来,并且隐式地包含了对该任务的因果假设和不对称性(即,不能在不改变语义的情况下交换标签)。MIL 问题出现在医疗保健(一个恶性细胞即表明癌症)、网络安全(一个恶意可执行文件即表明计算机被感染)以及许多其他任务中。在这项工作中,我们考察了五个最突出的深度 MIL 模型,发现它们中没有一个遵循标准的 MIL 假设。它们能够学习到反相关的实例,即默认输出“正”标签,直到看到一个负的反例,而对于一个正确的 MIL 模型来说,这种情况不应发生。我们怀疑由这些模型衍生出的增强方法和其他工作也会存在相同的问题。在任何使用这些模型的情境中,这都会造成学习到错误模型的潜在可能,从而带来操作失败的风险。我们通过一个提出的“算法单元测试”来识别并展示这一问题,在该测试中,我们创建了可由遵循 MIL 假设的模型解决的合成数据集,并且这些数据集能清晰地揭示违反 MIL 假设的学习行为。被评估的五个方法各自未能通过其中一项或多项测试。这提供了一种与模型无关的方法来识别建模假设的违反情况,我们希望这将对未来 MIL 模型的开发和评估有用。

英文摘要:

Multiple Instance Learning (MIL) is a sub-domain of classification problems with positive and negative labels and a "bag" of inputs, where the label is positive if and only if a positive element is contained within the bag, and otherwise is negative. Training in this context requires associating the bag-wide label to instance-level information, and implicitly contains a causal assumption and asymmetry to the task (i.e., you can't swap the labels without changing the semantics). MIL problems occur in healthcare (one malignant cell indicates cancer), cyber security (one malicious executable makes an infected computer), and many other tasks. In this work, we examine five of the most prominent deep-MIL models and find that none of them respects the standard MIL assumption. They are able to learn anti-correlated instances, i.e., defaulting to "positive" labels until seeing a negative counter-example, which should not be possible for a correct MIL model. We suspect that enhancements and other works derived from these models will share the same issue. In any context in which these models are being used, this creates the potential for learning incorrect models, which creates risk of operational failure. We identify and demonstrate this problem via a proposed "algorithmic unit test", where we create synthetic datasets that can be solved by a MIL respecting model, and which clearly reveal learning that violates MIL assumptions. The five evaluated methods each fail one or more of these tests. This provides a model-agnostic way to identify violations of modeling assumptions, which we hope will be useful for future development and evaluation of MIL models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑