GermanPartiesQA: 对商业大语言模型和AI伴侣在政治倾向和趋炎附势方面的基准测试
GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy
AI总结：
GermanPartiesQA研究了六个商业LLMs在政治倾向和趋炎附势方面的表现，揭示了LLMs在生成事实性政党立场上的局限性以及模型特定的意识形态倾向模式。
摘要：
大语言模型（LLMs）正在日益塑造公民的信息生态系统。整合LLMs的产品，如聊天机器人和AI伴侣，现在被广泛用于决策支持和信息检索，包括在敏感领域，这引发了对隐藏偏见和塑造个人决策和公众意见潜力的担忧。本文介绍了GermanPartiesQA，这是一个包含来自11次选举的418个政治声明的基准测试，用于评估六个商业LLMs。我们通过政治人设的角色扮演实验来评估它们的政治倾向。我们的评估揭示了三个具体发现：（1）事实限制：LLMs在准确生成事实性政党立场方面能力有限，尤其是对中间派政党。 （2）模型特定的意识形态倾向：我们识别了每个模型在不同温度设置和实验中的一致倾向模式和政治操控性程度。 （3）趋炎附势的声明：虽然模型在角色扮演中会调整到政治人设，但我们发现这反映的是基于人设的操控性，而非日益流行但存在争议的趋炎附势概念。我们的研究有助于评估日益嵌入选举决策支持工具和AI伴侣聊天机器人中的闭源LLMs的政治倾向。
英文摘要：
Large language models (LLMs) are increasingly shaping citizens' information ecosystems. Products incorporating LLMs, such as chatbots and AI Companions, are now widely used for decision support and information retrieval, including in sensitive domains, raising concerns about hidden biases and growing potential to shape individual decisions and public opinion. This paper introduces GermanPartiesQA, a benchmark of 418 political statements from German Voting Advice Applications across 11 elections to evaluate six commercial LLMs. We evaluate their political alignment based on role-playing experiments with political personas. Our evaluation reveals three specific findings: (1) Factual limitations: LLMs show limited ability to accurately generate factual party positions, particularly for centrist parties. (2) Model-specific ideological alignment: We identify consistent alignment patterns and the degree of political steerability for each model across temperature settings and experiments. (3) Claim of sycophancy: While models adjust to political personas during role-play, we find this reflects persona-based steerability rather than the increasingly popular, yet contested concept of sycophancy. Our study contributes to evaluating the political alignment of closed-source LLMs that are increasingly embedded in electoral decision support tools and AI Companion chatbots.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
