arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.04992cs.CL
收藏

ThaiSafetyBench: 评估泰语文化背景下语言模型的安全性

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

SCB DataX(SCB数据X);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);SCBX R&D(SCBX研发部);SCB 10X

Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul, Pakhapoom Sarapat

AI总结:

本研究提出ThaiSafetyBench,通过泰语文化背景下的恶意提示评估语言模型安全性,发现闭源模型安全性更强,同时揭示了文化特定攻击的高风险。

摘要:

大型语言模型(LLMs)的安全性评估仍然主要集中在英语上,导致非英语语言和基于文化的风险被忽视。在本工作中,我们研究了泰语语言和文化背景下LLM的安全性,并介绍了ThaiSafetyBench,一个包含1954个泰语恶意提示的开源基准测试集。该数据集涵盖了通用有害提示和明确基于泰语文化、社会和情境细节点的攻击。使用ThaiSafetyBench,我们评估了24个LLMs,其中GPT-4.1和Gemini-2.5-Pro作为LLM-as-a-judge评估者。我们的结果表明,闭源模型在安全性表现上通常优于开源模型,这引发了对公开可用模型鲁棒性的重大担忧。此外,我们观察到针对泰语特定、文化情境化的攻击的攻击成功率为比一般泰语攻击更高,突显了当前安全对齐方法的关键漏洞。为了提高可重复性和成本效率,我们进一步微调了一个基于DeBERTa的有害响应分类器,将其命名为ThaiSafetyClassifier。该模型实现了加权F1分数为84.4%,与GPT-4.1的判断相匹配。我们公开发布了微调权重和训练脚本以支持可重复性。最后,我们引入了ThaiSafetyBench排行榜,以提供持续更新的安全性评估并鼓励社区参与。- ThaiSafetyBench HuggingFace数据集:https://huggingface.co/datasets/typhoon-ai/ThaiSafetyBench - ThaiSafetyBench GitHub:https://github.com/trapoom555/ThaiSafetyBench - ThaiSafetyClassifier HuggingFace模型:https://huggingface.co/typhoon-ai/ThaiSafetyClassifier - ThaiSafetyBench排行榜:https://huggingface.co/spaces/typhoon-ai/ThaiSafetyBench-Leaderboard

英文摘要:

The safety evaluation of large language models (LLMs) remains largely centered on English, leaving non-English languages and culturally grounded risks underexplored. In this work, we investigate LLM safety in the context of the Thai language and culture and introduce ThaiSafetyBench, an open-source benchmark comprising 1,954 malicious prompts written in Thai. The dataset covers both general harmful prompts and attacks that are explicitly grounded in Thai cultural, social, and contextual nuances. Using ThaiSafetyBench, we evaluate 24 LLMs, with GPT-4.1 and Gemini-2.5-Pro serving as LLM-as-a-judge evaluators. Our results show that closed-source models generally demonstrate stronger safety performance than open-source counterparts, raising important concerns regarding the robustness of openly available models. Moreover, we observe a consistently higher Attack Success Rate (ASR) for Thai-specific, culturally contextualized attacks compared to general Thai-language attacks, highlighting a critical vulnerability in current safety alignment methods. To improve reproducibility and cost efficiency, we further fine-tune a DeBERTa-based harmful response classifier, which we name ThaiSafetyClassifier. The model achieves a weighted F1 score of 84.4%, matching GPT-4.1 judgments. We publicly release the fine-tuning weights and training scripts to support reproducibility. Finally, we introduce the ThaiSafetyBench leaderboard to provide continuously updated safety evaluations and encourage community participation. - ThaiSafetyBench HuggingFace Dataset: https://huggingface.co/datasets/typhoon-ai/ThaiSafetyBench - ThaiSafetyBench Github: https://github.com/trapoom555/ThaiSafetyBench - ThaiSafetyClassifier HuggingFace Model: https://huggingface.co/typhoon-ai/ThaiSafetyClassifier - ThaiSafetyBench Leaderboard: https://huggingface.co/spaces/typhoon-ai/ThaiSafetyBench-Leaderboard

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑