PRISM：一种审计大型语言模型偏见的方法论
PRISM: A Methodology for Auditing Biases in Large Language Models
AI总结：
本文提出PRISM，一种基于任务质询间接审计LLM偏见的灵活方法论，通过Political Compass Test对7个提供商的21个模型进行政治倾向评估，有效揭示了模型的默认立场、顺从度差异及限制空间。
摘要：
审计大型语言模型（LLMs）以发现其偏见和偏好，是创建负责任人工智能（AI）过程中的一项新兴挑战。尽管已有多种方法被提出用于引出此类模型的偏好，但LLM训练者已采取对策，使得LLM隐藏、混淆或直接拒绝披露其在某些特定主题上的立场。本文提出了PRISM，一种灵活的、基于质询的LLM审计方法论——该方法旨在通过基于任务的质询提示间接引出此类立场，而不是直接质询上述偏好。为了展示该方法的实用性，我们将PRISM应用于Political Compass Test，在其中评估了来自7个提供商的21个LLM的政治倾向。我们表明，LLM默认支持经济左倾和社会自由派的立场（与先前的研究一致）。我们还展示了这些模型愿意支持的立场空间——其中一些模型比其他模型更受限制且顺从度较低——而另一些则更加中立和客观。总之，PRISM能更可靠地探究和审计LLM，以理解其偏好、偏见和限制。
英文摘要：
Auditing Large Language Models (LLMs) to discover their biases and preferences is an emerging challenge in creating Responsible Artificial Intelligence (AI). While various methods have been proposed to elicit the preferences of such models, countermeasures have been taken by LLM trainers, such that LLMs hide, obfuscate or point blank refuse to disclosure their positions on certain subjects. This paper presents PRISM, a flexible, inquiry-based methodology for auditing LLMs - that seeks to illicit such positions indirectly through task-based inquiry prompting rather than direct inquiry of said preferences. To demonstrate the utility of the methodology, we applied PRISM on the Political Compass Test, where we assessed the political leanings of twenty-one LLMs from seven providers. We show LLMs, by default, espouse positions that are economically left and socially liberal (consistent with prior work). We also show the space of positions that these models are willing to espouse - where some models are more constrained and less compliant than others - while others are more neutral and objective. In sum, PRISM can more reliably probe and audit LLMs to understand their preferences, biases and constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
