审判中的公正：揭示（隐藏）在LLM辅助同行评审中的偏见
Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews
AI总结：
本文研究LLM生成的同行评审中的偏见，通过操控作者元数据发现机构偏见、资历偏好和性别影响，揭示隐性偏见在软评分中的显现。
摘要：
大型语言模型（LLM）的采用正在改变同行评审过程，从帮助评审员撰写详细评估到自动生成整个评审。尽管这些能力提供了新机会，但也引发了公平性和可靠性的担忧。本文通过操控作者元数据（包括隶属关系、性别、资历和出版历史）来研究LLM生成的同行评审中的偏见。我们的分析一致显示，高排名机构的作者存在强烈的隶属偏见。我们还识别出与资历和先前出版记录相关的方向性偏好，这些偏好可能影响边缘论文的接受决定。性别效应较小但在某些模型中存在。值得注意的是，隐性偏见在检查token级软评分时更加明显，表明对齐可能掩盖但无法完全消除潜在偏好。
英文摘要：
The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing detailed evaluations to generating entire reviews automatically. While these capabilities offer new opportunities, they also raise concerns about fairness and reliability. In this paper, we investigate bias in LLM-generated peer reviews through controlled interventions on author metadata, including affiliation, gender, seniority, and publication history. Our analysis consistently shows a strong affiliation bias favoring authors from highly ranked institutions. We also identify directional preferences associated with seniority and prior publication record, which can influence acceptance decisions for borderline papers. Gender effects are smaller but present in several models. Notably, implicit biases become more pronounced when examining token-level soft ratings, suggesting that alignment may mask but not fully eliminate underlying preferences
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
