超越黑盒标签：可解释的诊断标准用于主观NLP任务
Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks
AI总结：
本文提出一种在确定金标签前审计注释方案的可解释诊断方法，用于识别主观NLP任务中分歧的根源，通过分析多注释者标准判断，发现分歧集中在少数标准上，且近半数句子激活多个类别。
摘要：
主观NLP数据集通常将注释者判断汇总为单一金标签，难以判断分歧是否反映标准不明确、区分模糊或合理多样性。本文提出一种在确定金标签前审计专家设计的注释方案的可解释诊断方法，仅使用多注释者标准判断。该诊断区分两种失败模式：标准不稳定且边界难以操作化，以及系统重叠导致互斥类别的边界模糊。应用于商业文档中的说服价值提取，发现分歧并非分散：不稳定性集中在少数标准上，而近半数覆盖句子激活多个类别。这些信号与领域专家分歧一致，为收紧指南、修订类别结构或重新考虑注释范式提供证据支持。
英文摘要：
Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a \emph{schema-level diagnostic} for auditing expert-designed annotation schemas \emph{prior to} gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
