与调查受访者还是训练数据更一致？揭示LLMs在美国最高法院案例上的政治倾向
Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases
AI总结：
本文研究了LLMs的政治倾向更贴近预训练语料库还是调查中的人类意见，提出定量评估方法并在32个美国最高法院案例上验证，发现LLMs强烈反映训练数据中的政治倾向，与人类意见无强相关，强调了负责任整理训练数据和审计记忆的重要性。
摘要：
近期研究表明，大型语言模型（LLMs）倾向于记忆其训练数据中的模式和偏见，这引发了关于此类记忆内容如何影响模型行为的重要问题。其中之一便是LLM输出中政治偏见的出现。本文中，我们调查了LLMs的政治倾向在多大程度上反映了其预训练语料库中记忆的模式。我们提出了一种定量评估大型预训练语料库中嵌入的政治倾向的方法。随后，我们调查了LLMs的政治倾向更倾向于与谁一致：是其预训练语料库，还是被调查的人类意见。作为案例研究，我们重点探究了LLMs在32个美国最高法院案例中的政治倾向，涉及堕胎和投票权等争议性话题。我们的发现表明，LLMs强烈反映了其训练数据中的政治倾向，且未观察到其与调查中表达的人类意见的一致性存在强相关性。这些结果强调了负责任地整理训练数据的重要性，以及审计LLMs中记忆以确保人类与AI对齐的方法论。
英文摘要：
Recent works have shown that Large Language Models (LLMs) have a tendency to memorize patterns and biases present in their training data, raising important questions about how such memorized content influences model behavior. One such concern is the emergence of political bias in LLM outputs. In this paper, we investigate the extent to which LLMs' political leanings reflect memorized patterns from their pretraining corpora. We propose a method to quantitatively evaluate political leanings embedded in the large pretraining corpora. Subsequently we investigate to whom are the LLMs' political leanings more aligned with, their pretrainig corpora or the surveyed human opinions. As a case study, we focus on probing the political leanings of LLMs in 32 US Supreme Court cases, addressing contentious topics such as abortion and voting rights. Our findings reveal that LLMs strongly reflect the political leanings in their training data, and no strong correlation is observed with their alignment to human opinions as expressed in surveys. These results underscore the importance of responsible curation of training data, and the methodology for auditing the memorization in LLMs to ensure human-AI alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
