arXiv 2506.051422025-06-09cs.CL

大型语言模型会像人类一样判断错误严重程度吗？

Do Large Language Models Judge Error Severity Like Humans?

School of Psychology（心理学系）；Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning（湖北省人工智能与智能学习重点实验室）；National Language Resources Monitor and Research Center for Network Media（网络媒体语言资源监测与研究中心）；School of Computer Science（计算机科学学院）

Diege Sun, Guanyi Chen, Zhao Fan, Xiaorong Cheng, Tingting He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过受控图像描述错误比较人类与LLM的严重程度判断，发现多数模型偏差明显，DeepSeek-V3对齐最佳，Doubao唯一复现人类排序但区分度不足。

摘要：

大型语言模型（LLM）正越来越多地被用作自然语言生成中的自动评估器，但它们能否准确复现人类对错误严重程度的判断仍不清楚。在本研究中，我们系统比较了人类与LLM对包含受控语义错误的图像描述的评估。我们将van Miltenburg等人（2020）的实验框架扩展到单模态（仅文本）和多模态（文本＋图像）两种设置，评估四类错误：年龄、性别、服装类型和服装颜色。研究结果表明，人类会为不同错误类型赋予不同的严重程度，而视觉情境会显著放大颜色错误和类型错误的感知严重程度。值得注意的是，大多数LLM给性别错误较低分数，却给颜色错误不成比例的高分；这与人类不同，人类认为两者都非常严重，但原因不同。这表明这些模型可能内化了影响性别判断的社会规范，却缺乏模拟人类对颜色敏感性所需的知觉基础，而这种敏感性由不同的神经机制塑造。在受评估的LLM中，仅有Doubao复现了类似人类的错误严重程度排序，但它未能像人类那样清晰地区分错误类型。令人惊讶的是，单模态LLM DeepSeek-V3在单模态和多模态条件下均取得了与人类判断最高的一致性，甚至优于最先进的多模态模型。

英文摘要：

Large Language Models (LLMs) are increasingly used as automated evaluators in natural language generation, yet it remains unclear whether they can accurately replicate human judgments of error severity. In this study, we systematically compare human and LLM assessments of image descriptions containing controlled semantic errors. We extend the experimental framework of van Miltenburg et al. (2020) to both unimodal (text-only) and multimodal (text + image) settings, evaluating four error types: age, gender, clothing type, and clothing colour. Our findings reveal that humans assign varying levels of severity to different error types, with visual context significantly amplifying perceived severity for colour and type errors. Notably, most LLMs assign low scores to gender errors but disproportionately high scores to colour errors, unlike humans, who judge both as highly severe but for different reasons. This suggests that these models may have internalised social norms influencing gender judgments but lack the perceptual grounding to emulate human sensitivity to colour, which is shaped by distinct neural mechanisms. Only one of the evaluated LLMs, Doubao, replicates the human-like ranking of error severity, but it fails to distinguish between error types as clearly as humans. Surprisingly, DeepSeek-V3, a unimodal LLM, achieves the highest alignment with human judgments across both unimodal and multimodal conditions, outperforming even state-of-the-art multimodal models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。