Pak3H：使用人类语境化乌尔都语基准评估LLM对齐中的文化不匹配成本
Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark
浏览论文内容
中文总结 AI 辅助
该研究针对LLM对齐中低资源语言的文化不匹配问题，构建了首个人类验证的乌尔都语3H对齐基准Pak3H1，经评估发现多语言LLM存在对齐差距，凸显人工引导本地化评估的必要性。
中文摘要 AI 辅助
大语言模型（LLM）在以英语为中心的场景中展现出强大的有用性、无害性和诚实性（3H）对齐能力，但由于文化不匹配，这些成果向低资源语言的迁移效果很差。现有的多语言3H基准主要依赖自动翻译或基于LLM的合成，会传播源语言偏见并牺牲本地相关性。为解决这一差距，我们推出Pak3H1，这是首个经人工验证、文化语境化的乌尔都语3H对齐基准套件，包含PakAlpaca（有用性）、PakBeaverTails（无害性）和PakTruthfulQA（诚实性）。我们的多阶段流程整合了人工文化适配和词典引导的后期编辑，优先考虑母语使用者的判断，确保语义保真度和语境真实性。对多个开源和专有LLM架构的零样本评估显示出系统性的跨语言对齐差距：在本地化语境下，有用性胜率下降；无害性防护措施在应对区域安全风险时失效；受本地化事实约束，综合诚实性指标大幅下降。这些发现揭示了当前对齐方法的结构性局限，强调了人工引导的本地化对于公平多语言评估的必要性。
英文摘要
Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains transfer poorly to low-resource languages due to cultural mismatches. Existing multilingual 3H benchmarks rely predominantly on automated translation or LLM based synthesis, propagating source-language biases while sacrificing local relevance. To address this gap, we introduce Pak3H1, the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty). Our multi-stage pipeline integrates manual cultural adaptation and dictionary-guided post editing to prioritize native speaker judgment, ensuring both semantic fidelity and contextual authenticity. Zero-shot evaluations across multiple open and proprietary LLM architectures reveal systematic cross-lingual alignment gaps: helpfulness win rates decline under localized contexts, harmlessness guardrails break down against regional safety risks, and composite honesty metrics degrade substantially due to localized factual constraints. These findings expose structural limitations in current alignment approaches, underscoring the necessity of human-guided localization for equitable multilingual evaluation.
发表机构
- Macquarie University（麦考瑞大学）
- Lahore University of Management Sciences（拉合尔管理科学大学）
机构由 AI 辅助整理，请以论文原文为准。
