基于实例级Fano界的人类标签变异情绪可预测性的偏差校正上限
Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds
查看机构详情
- Faculty of Business and Commerce, Kansai University（关西大学商学部）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
本文提出BACE框架，结合实例级Fano界等方法，量化文本情感识别性能上限的影响因素，发现GoEmotions等任务中分类器至少约33%误差为不可约误差，无法通过单一估计器判定性能是否饱和。
中文摘要 AI 辅助
文本情感识别在基准测试上的性能不断提升，但很少有人严谨地探究是否已达到准确率上限。本文的目标并非将该上限确定为单一数值，而是量化其在多大程度上取决于有限标注、估计器选择、标注噪声和评估协议，从而为断言性能饱和提供依据。我们提出了偏差校正情感上限估计框架（BACE），该框架可估计偏差校正后的上限，分离不可约误差与可约误差，并规范相关断言。其中，锚定狄利克雷混合经验贝叶斯估计器介于插件估计器和NSB估计器之间，可恢复人类共识分布；通过标注者拆分、噪声反卷积和固定断言门，可无循环地归因误差。方法层面，无约束点估计将可达性范围设定为0.38至1.03，因此无法通过单一估计器判定是否达到饱和；实证层面，通过断言门的唯一结论是，代表性分类器在GoEmotions数据集上的误差中至少约33%为不可约误差，且该模式在冒犯性和反讽任务中重复出现。
英文摘要
Emotion recognition from text keeps improving on benchmarks, yet whether an accuracy ceiling has been reached is seldom asked with discipline. Our aim is not to pin this ceiling to a single number, but to quantify how far it depends on finite annotation, estimator choice, annotation noise, and the evaluation protocol, and thereby to discipline how confidently saturation can be claimed. We propose Bias-corrected Affective Ceiling Estimation (BACE), an analysis framework that estimates a bias-corrected ceiling, separates irreducible from reducible error, and disciplines the resulting claims. An anchored Dirichlet-mixture empirical Bayes estimator, bracketed between plug-in and NSB, recovers the human-consensus distribution; an annotator split, a noise deconvolution, and a fixed claim gate then attribute error without circularity. Methodologically, unconstrained point estimates place reachability anywhere from 0.38 to 1.03, so saturation cannot be decided by any single estimator. Substantively, the only assertion passing the claim gate is that at least about 33% of a representative classifier's error on GoEmotions is irreducible, with the same pattern recurring on offensiveness and irony.
