对政治倾向大型语言模型的多维审计
A Multi-Dimensional Audit of Politically Aligned Large Language Models
AI总结：
本文提出基于哈贝马斯沟通行动理论的多维框架，评估政治倾向语言模型在有效性、公平性、真实性及说服力四方面的表现，揭示大模型在政治角色扮演中更有效但更偏颇的权衡问题。
摘要：
随着大型语言模型（LLMs）在各行业中的应用日益广泛，对其潜在滥用的担忧，尤其是在政治讨论领域日益增加。通过提示工程或微调技术有意使LLMs与特定政治意识形态对齐，在政治竞选等用例中具有优势，但需谨慎考虑因性能退化、虚假信息或偏见行为增加而带来的风险。本文提出受哈贝马斯沟通行动理论启发的多维框架，通过自动化定量指标在四个维度上审计政治倾向语言模型：有效性、公平性、真实性及说服力。对九种通过微调或角色扮演对齐的流行LLM应用此框架，揭示出一致的权衡：虽然较大模型在扮演政治意识形态和回应真实性方面更有效，但它们在公平性方面也更差，表现为对不同意识形态的人使用愤怒和有毒语言。微调模型相比角色扮演模型表现出更低的偏见和更有效的对齐，但同时也在推理任务上表现下降且幻觉增加。总体而言，所有测试模型在至少一个四维指标上都存在缺陷，突显了需要更平衡和稳健的对齐策略的必要性。最终，本文旨在确保政治倾向LLMs生成合法、无害的论点，提供一个评估这些模型负责任政治对齐的框架。
英文摘要：
As the application of Large Language Models (LLMs) spreads across various industries, there are increasing concerns about the potential for their misuse, especially in sensitive areas such as political discourse. Deliberately aligning LLMs with specific political ideologies, through prompt engineering or fine-tuning techniques, can be advantageous in use cases such as political campaigns, but requires careful consideration due to heightened risks of performance degradation, misinformation, or increased biased behavior. In this work, we propose a multi-dimensional framework inspired by Habermas' Theory of Communicative Action to audit politically aligned language models across four dimensions: effectiveness, fairness, truthfulness, and persuasiveness using automated, quantitative metrics. Applying this to nine popular LLMs aligned via fine-tuning or role-playing revealed consistent trade-offs: while larger models tend to be more effective at role-playing political ideologies and truthful in their responses, they were also less fair, exhibiting higher levels of bias in the form of angry and toxic language towards people of different ideologies. Fine-tuned models exhibited lower bias and more effective alignment than the corresponding role-playing models, but also saw a decline in performance reasoning tasks and an increase in hallucinations. Overall, all of the models tested exhibited some deficiency in at least one of the four metrics, highlighting the need for more balanced and robust alignment strategies. Ultimately, this work aims to ensure politically-aligned LLMs generate legitimate, harmless arguments, offering a framework to evaluate the responsible political alignment of these models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
