探戈需要两人共舞：导航 NLP 任务的概念化与性能测量的分歧
It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance
AI总结：
本文基于测量建模工具提出了一种区分任务概念化与性能测量操作化的NLP基准分歧分类法，通过元分析和八项任务的从业者调查验证了该分类法，并据此提出了构建基准及记录其局限性的框架。
摘要：
NLP 的进展日益通过基准来衡量；因此，将进展置于语境中需要理解从业者何时以及为何会对基准的有效性产生分歧。我们利用测量建模的工具开发了一套分歧分类法，并区分了两种分歧类型：1）任务如何被概念化，以及 2）模型性能的测量如何被操作化。为了给我们的分类法提供证据，我们对相关文献进行了元分析以理解 NLP 任务如何被概念化，同时调查了从业者对不同影响基准有效性因素的印象。我们的元分析和调查涵盖了从共指消解到问答的八项任务，发现任务通常没有被清晰且一致地概念化，且基准存在操作化分歧。这些发现支持了我们提出的分歧分类法。最后，基于我们的分类法，我们提出了一个构建基准并记录其局限性的框架。
英文摘要：
Progress in NLP is increasingly measured through benchmarks; hence, contextualizing progress requires understanding when and why practitioners may disagree about the validity of benchmarks. We develop a taxonomy of disagreement, drawing on tools from measurement modeling, and distinguish between two types of disagreement: 1) how tasks are conceptualized and 2) how measurements of model performance are operationalized. To provide evidence for our taxonomy, we conduct a meta-analysis of relevant literature to understand how NLP tasks are conceptualized, as well as a survey of practitioners about their impressions of different factors that affect benchmark validity. Our meta-analysis and survey across eight tasks, ranging from coreference resolution to question answering, uncover that tasks are generally not clearly and consistently conceptualized and benchmarks suffer from operationalization disagreements. These findings support our proposed taxonomy of disagreement. Finally, based on our taxonomy, we present a framework for constructing benchmarks and documenting their limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
