重新审视真实标签：在大语言模型基准测试中的人类标签变异案例研究
Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking
AI总结：
研究探讨了人类标注差异对大语言模型基准测试的影响，发现大模型在高一致性子集表现最佳，但在高分歧情况下表现不佳，表明参数数量不决定对模糊性和主观性的敏感度。
摘要：
人类标签变异（HLV），即标注者判断间的系统性差异，在基准测试中仍被忽视，尽管大语言模型（LLM）发展迅速。我们通过引入一种多模态大语言模型（MLLM）基准测试评估协议来填补这一空白，该协议明确考虑了两个条件：（1）人类标签一致性和（2）分歧。我们应用此协议对两个最先进的MLLM家族（Gemma 3、Qwen 2.5 VL）进行测试，使用非聚合的人类标注数据集进行社交媒体内容分类。在各类任务中，我们发现较大模型在高一致性子集表现最佳，但在高分歧情况下往往表现不佳，表明参数数量本身并不决定对模糊性和主观性的敏感度。这些结果表明，仅基于共识标签的基准测试可能夸大模型在该领域的能力，而纳入人类标签变异可提供更现实和稳健的MLM在内容审核流水线中的评估。
英文摘要：
Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
