迷失于定位：通过人类在环验证构建RabakBench以衡量多语言安全差距
Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps
AI总结：
RabakBench通过人类在环验证构建，用于衡量多语言安全差距，通过三阶段流程生成、标注和翻译数据，评估LLM在低资源语言中的安全性问题。
摘要：
大型语言模型（LLMs）在低资源语言变体中往往难以维持安全性，例如混合语言的口语和方言。我们引入了RabakBench，这是一个多语言安全基准和可扩展的本地化流程，专门针对新加坡独特的语言景观，涵盖Singlish、中文、马来语和泰米尔语。我们通过三阶段流程构建该基准：（1）生成：通过LLM驱动的红队行动增强现实中的不安全网络内容；（2）标注：使用多数投票的LLM标注器进行半自动多标签标注；（3）翻译：进行高保真度且保留毒性特征的翻译。所得到的数据集包含超过5000个示例，覆盖六个细粒度的安全类别。尽管使用LLM实现可扩展性，我们的框架仍保持严格的人员监督，达到0.70-0.80的标注者间一致性。对13种最先进的防护措施进行评估，揭示了显著的性能下降，凸显了本地化评估的必要性。RabakBench提供了一个可重复的框架，用于在未被服务的社区中构建安全基准。
英文摘要：
Large language models (LLMs) often fail to maintain safety in low-resource language varieties, such as code-mixed vernaculars and regional dialects. We introduce RabakBench, a multilingual safety benchmark and scalable pipeline localized to Singapore's unique linguistic landscape, covering Singlish, Chinese, Malay, and Tamil. We construct the benchmark through a three-stage pipeline: (1) Generate: augmenting real-world unsafe web content via LLM-driven red teaming; (2) Label: applying semi-automated multi-label annotation using majority-voted LLM labelers; and (3) Translate: performing high-fidelity, toxicity-preserving translation. The resulting dataset contains over 5,000 examples across six fine-grained safety categories. Despite using LLMs for scalability, our framework maintains rigorous human oversight, achieving 0.70-0.80 inter-annotator agreement. Evaluations of 13 state-of-the-art guardrails reveal significant performance degradation, underscoring the need for localized evaluation. RabakBench provides a reproducible framework for building safety benchmarks in underserved communities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
