标签效应：人类和LLM-as-a-Judge在信任评估中的共享启发式依赖
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
AI总结：
研究揭示人类和LLM在信任评估中受标签影响的共同机制，通过对比人类和模型的反应，发现标签对信任判断有显著偏倚，提出需警惕标签敏感的LLM评估有效性。
摘要：
大型语言模型（LLMs）日益被用作自动评估者（LLM-as-a-Judge）。本工作挑战其可靠性，通过展示LLM的信任判断受披露源标签的偏倚，发现人类和LLM评估者均更信任标记为人生成的内容而非AI生成的内容。眼动数据揭示人类严重依赖源标签作为判断的启发式线索。分析LLM内部状态时发现，不同标签条件下，模型在标签区域分配更密集的注意力，且在人类标签下比AI标签更明显，与人类目光模式一致。此外，logits测得的决策不确定性在AI标签下高于人类标签。这些结果表明，源标签是人类和LLM的显著启发式线索。这引发了标签敏感LLM-as-a-Judge评估的有效性问题，我们谨慎指出对齐模型与人类偏好可能将人类启发式依赖传播至模型，推动去偏评估和对齐。
英文摘要：
Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge). This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels. Using a counterfactual design, we find that both humans and LLM judges assign higher trust to information labeled as human-authored than to the same content labeled as AI-generated. Eye-tracking data reveal that humans rely heavily on source labels as heuristic cues for judgments. We analyze LLM internal states during judgment. Across label conditions, models allocate denser attention to the label region than the content region, and this label dominance is stronger under Human labels than AI labels, consistent with the human gaze patterns. Besides, decision uncertainty measured by logits is higher under AI labels than Human labels. These results indicate that the source label is a salient heuristic cue for both humans and LLMs. It raises validity concerns for label-sensitive LLM-as-a-Judge evaluation, and we cautiously raise that aligning models with human preferences may propagate human heuristic reliance into models, motivating debiased evaluation and alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
