SomaliBench Eval：衡量开源语言模型中英语到索马里语的拒绝差距
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
AI总结：
通过构建索马里语有害意图基准并评估四个开源模型，发现英语到索马里语的拒绝率存在显著差距，且多数非拒绝输出为不流畅的无效内容。
摘要：
大型语言模型的安全评估仍然高度以英语为中心，即使模型在全球部署，低资源语言的评估也严重不足。我们在SomaliBench v0上评估了四个开源指令微调模型，这是一个由母语者验证的基准，包含100对英语和索马里语的有害意图提示。每个模型（Llama-3.1-8B-Instruct、Gemma-2-9B-Instruct、Qwen-2.5-7B-Instruct和Aya-23-8B）均在本地运行，温度为0，并使用相同的英语“有帮助、无害、诚实”（HHH）系统提示。一个固定的Claude Sonnet快照（claude-sonnet-4-5-20250929）将每个响应分类为拒绝、遵从或不清楚；母语作者对分层抽样的80行样本进行抽查。我们发现所有四个模型在英语到索马里语之间存在巨大的拒绝差距：Llama-3.1-8B（0.90；95%自助法置信区间[0.85, 0.96]）、Aya-23-8B（0.75 [0.67, 0.83]）、Qwen-2.5-7B（0.69 [0.59, 0.78]）和Gemma-2-9B（0.38 [0.27, 0.49]）。对于三个模型，索马里语中主要的非拒绝模式不是流畅的有害遵从，而是不清楚的输出：空、错误语言或不连贯的生成。母语验证抽查在80个采样行上与判断器达到100%一致（Cohen's kappa = 1.00）。我们仅报告总体拒绝率、类别差距和可靠性统计；原始模型生成保留在本地，不发布。
英文摘要：
Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed globally. We evaluate four open-weight instruction-tuned models on SomaliBench v0, a native-author-verified benchmark of 100 harmful-intent prompts paired across English and Somali. Each of Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, Qwen-2.5-7B-Instruct, and Aya-23-8B is run locally with temperature 0 and the same English "helpful, harmless, and honest" (HHH) system prompt. We find large English-to-Somali refusal gaps for all four models, ranging from 0.40 to 0.93, all strictly positive under a paired bootstrap and significant by exact McNemar tests. For three models, the dominant Somali non-refusal mode is not fluent harmful compliance but unclear output: wrong-language, incoherent, or off-topic generations. A pinned Claude Sonnet snapshot (claude-sonnet-4-5-20250929) classifies each response as refused, complied, or unclear; its safety layer declined 34 of 800 classifications, which the native author labeled manually. A native-author spot-check achieves 100% agreement with the judge (Cohen's $κ=1.00$) on 74 comparable rows. We report aggregate refusal rates, category gaps, and reliability statistics only; raw model generations are retained locally and are not released because some may contain harmful content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
