超越准确性：通过测量误差一致性来量化CNN和人类的试验行为
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
摘要：
认知科学和行为神经科学以及机器学习和人工智能研究的一个核心问题是确定两个或多个决策者（无论是大脑还是算法）是否使用相同的策略。准确性本身并不能区分不同的策略：两个系统可以用非常不同的策略达到相似的准确性。如果两个系统的性能接近天花板，就像卷积神经网络（CNN）和人类在视觉物体识别上的表现一样，那么区分准确性之外的差异就显得尤为迫切。在这里，我们介绍了一次又一次的试错一致性，这是一种定量分析，用于衡量两个决策系统是否在相同的输入上系统地犯错误。在一次又一次尝试的基础上犯下一致的错误是决策者之间相似处理策略的必要条件。我们的分析适用于比较算法与算法，人类与人类，以及算法与人类。当错误一致性应用于物体识别时，我们得到了三个主要发现：（1）。无论架构如何，CNN彼此之间都非常一致。（2.）然而，CNN和人类观察者之间的一致性仅略高于偶然的预期-这表明人类和CNN可能实施非常不同的策略。（3.）CORnet-S是一种被称为“灵长类动物腹侧视觉流的当前最佳模型”的递归模型，它未能捕获人类行为数据的基本特征，并且在我们的分析中表现得基本上像一个标准的纯前馈ResNet-50。总的来说，错误一致性分析表明人类和机器视觉使用的策略仍然非常不同-但我们设想我们的通用错误一致性分析作为量化未来进展的有效工具。
英文摘要：
A central problem in cognitive science and behavioural neuroscience as well as in machine learning and artificial intelligence research is to ascertain whether two or more decision makers (be they brains or algorithms) use the same strategy. Accuracy alone cannot distinguish between strategies: two systems may achieve similar accuracy with very different strategies. The need to differentiate beyond accuracy is particularly pressing if two systems are near ceiling performance, like Convolutional Neural Networks (CNNs) and humans on visual object recognition. Here we introduce trial-by-trial error consistency, a quantitative analysis for measuring whether two decision making systems systematically make errors on the same inputs. Making consistent errors on a trial-by-trial basis is a necessary condition for similar processing strategies between decision makers. Our analysis is applicable to compare algorithms with algorithms, humans with humans, and algorithms with humans. When applying error consistency to object recognition we obtain three main findings: (1.) Irrespective of architecture, CNNs are remarkably consistent with one another. (2.) The consistency between CNNs and human observers, however, is little above what can be expected by chance alone -- indicating that humans and CNNs are likely implementing very different strategies. (3.) CORnet-S, a recurrent model termed the "current best model of the primate ventral visual stream", fails to capture essential characteristics of human behavioural data and behaves essentially like a standard purely feedforward ResNet-50 in our analysis. Taken together, error consistency analysis suggests that the strategies used by human and machine vision are still very different -- but we envision our general-purpose error consistency analysis to serve as a fruitful tool for quantifying future progress.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
