能力问题：关于发现和消除语言数据中的伪像
Competency Problems: On Finding and Removing Artifacts in Language Data
摘要：
NLP领域最近的许多工作都记录了数据集伪影、偏差以及输入特征和输出标签之间的虚假相关性。然而，如何区分哪些特征具有“虚假”而不是合法的相关性通常是未指定的。在这项工作中，我们认为，对于复杂的语言理解任务，所有简单的功能相关性是虚假的，我们正式将这一概念转化为一类问题，我们称之为能力问题。例如，“惊人”一词本身不应该给信息的情感标签独立的上下文中，它出现，其中可能包括否定，隐喻，讽刺等，我们从理论上分析的困难创建数据的能力问题时，考虑到人类的偏见，现实的数据集将越来越偏离能力问题，数据集大小的增加。这种分析为我们提供了一个简单的数据集工件统计测试，我们使用它来显示比以前的工作中描述的更微妙的偏差，包括证明模型受到这些不太极端的偏差的不适当影响。我们对这个问题的理论处理也使我们能够分析建议的解决方案，例如对数据集实例进行本地编辑，并为未来的数据汇集和针对能力问题的模型设计工作提供建议。
英文摘要：
Much recent work in NLP has documented dataset artifacts, bias, and spurious correlations between input features and output labels. However, how to tell which features have "spurious" instead of legitimate correlations is typically left unspecified. In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems. For example, the word "amazing" on its own should not give information about a sentiment label independent of the context in which it appears, which could include negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that realistic datasets will increasingly deviate from competency problems as dataset size increases. This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases. Our theoretical treatment of this problem also allows us to analyze proposed solutions, such as making local edits to dataset instances, and to give recommendations for future data collection and model design efforts that target competency problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
