量化大语言模型自评与互评中的标签诱导偏差
Quantifying Label-Induced Bias in Large Language Model Self- and Cross-Evaluations
AI总结：
本研究通过受控实验量化了ChatGPT、Gemini和Claude在自评与互评中因模型标签引发的系统性偏差，发现标签可显著扭曲评分与偏好，凸显了盲评协议和多模型验证的必要性。
摘要：
大语言模型（LLMs）正越来越多地被部署为文本质量的评估者，然而其判断的有效性仍未得到充分探索。本研究调查了三个知名LLM——ChatGPT、Gemini和Claude——在自评与互评中的系统性偏差。我们设计了一个受控实验，其中由每个模型撰写的博客文章被所有三个模型在四种标签条件下进行评估：无归属、真实归属以及两种错误归属场景。评估采用了整体偏好投票和跨三个维度（连贯性、信息性和简洁性）的细粒度质量评分，所有分数均归一化为百分比以便直接比较。我们的研究结果揭示了模型判断中的显著不对称性：“Claude”标签无论实际作者是谁都持续提升分数，而“Gemini”标签则系统性压低分数。错误归属经常逆转偏好排名，导致投票结果出现高达50个百分点的转变，质量评分出现高达12个百分点的转变。值得注意的是，Gemini在真实标签下表现出严重的自我贬低，而Claude则表现出强化的自我偏好。这些结果表明，感知到的模型身份可以大幅扭曲高层判断和细粒度质量评估，且与内容质量无关。我们的发现挑战了LLM作为评判者范式的可靠性，并强调了采用盲评协议和多样化多模型验证框架的迫切需要，以确保自动文本评估和LLM基准测试的公平性与有效性。
英文摘要：
Large language models (LLMs) are increasingly deployed as evaluators of text quality, yet the validity of their judgments remains underexplored. This study investigates systematic bias in self- and cross-model evaluations across three prominent LLMs: ChatGPT, Gemini, and Claude. We designed a controlled experiment in which blog posts authored by each model were evaluated by all three models under four labeling conditions: no attribution, true attribution, and two false-attribution scenarios. Evaluations employed both holistic preference voting and granular quality ratings across three dimensions Coherence, Informativeness, and Conciseness with all scores normalized to percentages for direct comparison. Our findings reveal pronounced asymmetries in model judgments: the "Claude" label consistently elevated scores regardless of actual authorship, while the "Gemini" label systematically depressed them. False attribution frequently reversed preference rankings, producing shifts of up to 50 percentage points in voting outcomes and up to 12 percentage points in quality ratings. Notably, Gemini exhibited severe self-deprecation under true labels, while Claude demonstrated intensified self-preference. These results demonstrate that perceived model identity can substantially distort both high-level judgments and fine-grained quality assessments, independent of content quality. Our findings challenge the reliability of LLM-as-judge paradigms and underscore the critical need for blind evaluation protocols and diverse multi-model validation frameworks to ensure fairness and validity in automated text evaluation and LLM benchmarking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
