关于大型语言模型在辩论评估中的实证分析
An Empirical Analysis on Large Language Models in Debate Evaluation
AI总结：
本研究探讨了GPT-3.5和GPT-4在辩论评估中的能力与偏见，发现其表现超越人类和微调模型，并揭示了位置、词汇和结束等固有偏见。
摘要：
在本研究中，我们探讨了GPT-3.5和GPT-4等先进大型语言模型（LLMs）在辩论评估中的能力和固有偏见。我们发现，LLMs的表现超越了人类，并且超越了在庞大数据集上微调的最先进方法的性能。我们进一步探索并分析了LLMs中存在的偏见，包括可能影响其评估判断的位置偏见、词汇偏见和顺序偏见。我们的发现表明，GPT-3.5和GPT-4对呈现的第二个候选回答存在一致的偏见，这归因于提示设计。我们还揭示了两者中的词汇偏见，特别是当标签集带有数字或顺序等含义时，突显了在提示设计中谨慎选择标签具象化词的必要性。此外，分析表明这两个模型都倾向于将辩论的总结方判定为获胜者，暗示存在讨论结束偏见。
英文摘要：
In this study, we investigate the capabilities and inherent biases of advanced large language models (LLMs) such as GPT-3.5 and GPT-4 in the context of debate evaluation. We discover that LLM's performance exceeds humans and surpasses the performance of state-of-the-art methods fine-tuned on extensive datasets in debate evaluation. We additionally explore and analyze biases present in LLMs, including positional bias, lexical bias, order bias, which may affect their evaluative judgments. Our findings reveal a consistent bias in both GPT-3.5 and GPT-4 towards the second candidate response presented, attributed to prompt design. We also uncover lexical biases in both GPT-3.5 and GPT-4, especially when label sets carry connotations such as numerical or sequential, highlighting the critical need for careful label verbalizer selection in prompt design. Additionally, our analysis indicates a tendency of both models to favor the debate's concluding side as the winner, suggesting an end-of-discussion bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
