arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.06263cs.CYcs.AIcs.CLcs.LG
收藏

关键外交政策决策(CFPD)基准:衡量大语言模型中的外交偏好

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models

Center for Strategic and International Studies(战略与国际研究中心);Scale AI(Scale AI公司)

Benjamin Jensen, Ian Reynolds, Yasir Atalan, Michael Garcia, Austin Woo, Anthony Chen, Trevor Howarth

AI总结:

本文提出 CFPD 基准,通过 400 个专家场景评估七个主流大语言模型在国际关系中的外交偏好,发现模型间存在显著差异和国别偏见,强调高风险场景下需进行领域化评估与微调。

摘要:

随着国家安全机构日益将人工智能(AI)整合到决策和内容生成过程中,理解大语言模型(LLMs)的内在偏见至关重要。本研究提出了一个新颖的基准,用于评估七个重要基础模型在国际关系(IR)背景下的偏见和偏好,这些模型包括 Llama 3.1 8B Instruct、Llama 3.1 70B Instruct、GPT-4o、Gemini 1.5 Pro-002、Mixtral 8x22B、Claude 3.5 Sonnet 和 Qwen2 72B。我们围绕国际关系的核心主题设计了一项偏见发现研究,使用 400 个专家构建的场景来分析所选模型的结果。这些场景聚焦于四个主题领域,包括:军事升级、军事和人道主义干预、国际体系中的合作行为以及联盟动态。我们的分析揭示了模型建议在四个测试领域场景中的显著差异。特别是,Qwen2 72B、Gemini 1.5 Pro-002 和 Llama 3.1 8B Instruct 模型给出的升级性建议显著多于 Claude 3.5 Sonnet 和 GPT-4o 模型。所有模型都表现出一定程度的国别偏见,与美国和英国相比,它们通常对中国和俄罗斯建议较少升级性和干预性的行动。这些发现强调了在高风险环境中受控部署大语言模型的必要性,并指出需要开展特定领域的评估和模型微调,以使其与机构目标保持一致。

英文摘要:

As national security institutions increasingly integrate Artificial Intelligence (AI) into decision-making and content generation processes, understanding the inherent biases of large language models (LLMs) is crucial. This study presents a novel benchmark designed to evaluate the biases and preferences of seven prominent foundation models-Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, GPT-4o, Gemini 1.5 Pro-002, Mixtral 8x22B, Claude 3.5 Sonnet, and Qwen2 72B-in the context of international relations (IR). We designed a bias discovery study around core topics in IR using 400-expert crafted scenarios to analyze results from our selected models. These scenarios focused on four topical domains including: military escalation, military and humanitarian intervention, cooperative behavior in the international system, and alliance dynamics. Our analysis reveals noteworthy variation among model recommendations based on scenarios designed for the four tested domains. Particularly, Qwen2 72B, Gemini 1.5 Pro-002 and Llama 3.1 8B Instruct models offered significantly more escalatory recommendations than Claude 3.5 Sonnet and GPT-4o models. All models exhibit some degree of country-specific biases, often recommending less escalatory and interventionist actions for China and Russia compared to the United States and the United Kingdom. These findings highlight the necessity for controlled deployment of LLMs in high-stakes environments, emphasizing the need for domain-specific evaluations and model fine-tuning to align with institutional objectives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑