差异的两个表情符号：多语言情感生成基准实际上衡量了什么
Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure
浏览论文内容
中文总结 AI 辅助
本研究审计多语言情感生成基准，发现其结论多为测量工具伪影，并提出基于参考的表情符号-情感可解码性探针，以提供更稳定的系统排名。
中文摘要 AI 辅助
我们对一个多语言情感生成基准进行了审计，涉及八个指令微调的大语言模型为17,100个孟加拉语、英语和印地语句子生成表情符号摘要，并使用了6,960个人工判断。我们发现其头条结论是测量工具的人为产物，而非系统本身的属性。将标注者视为随机效应而非固定效应时，没有任何系统与其他系统存在显著差异（F(7,14)=0.59，p=0.76），而传统分析却宣称28对差异中有19对显著。标注者身份解释的评分方差远多于系统身份，且移除任何单个标注者都会改变获胜系统。确实出现的排序与输出长度相关：平均表情符号数量解释了系统间方差的78.7%，而基于2,599对样本的条目内长度匹配比较则逆转了排行榜。我们进一步表明，跨提供者的各向异性差异在均值中心化后消失，每语言令牌成本随归一化单位改变符号，多视图行级拆分使宏F1分数虚增3.1个百分点并改变排名最高的系统。我们提出用表情符号-情感可解码性替代偏好评分，这是一种基于参考的探针，其排名在不同随机种子间对宏F1的稳定性达到±0.003。
英文摘要
We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other ($F(7,14)=0.59$, $p=0.76$), although the conventional analysis declares 19 of 28 pairwise differences significant. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed. The ordering that does emerge tracks output length: mean emoji count explains 78.7\% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. We further show that cross-provider anisotropy differences vanish under mean-centring, that per-language token costs change sign with the normalising unit, and that multi-view row-wise splits inflate macro-F1 by $3.1$ points and change the top-ranked system. In place of preference scoring we propose **emoji-affect decodability**, a reference-based probe whose rankings are stable to $\pm0.003$ macro-F1 across seeds.
发表机构
- Islamic University of Technology（伊斯兰理工大学）
机构由 AI 辅助整理，请以论文原文为准。
