大型语言模型中对抗性偏见诱导鲁棒性的基准测试：基于LLM-as-a-Judge的可扩展自动化评估
Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge
AI总结：
提出可扩展基准框架，用LLM-as-a-Judge评估LLM对抗性偏见诱导的鲁棒性，发布CLEAR-Bias数据集，发现偏见韧性不均且无模型完全鲁棒。
摘要：
大型语言模型（LLMs）在关键社会领域的日益整合引发了人们对嵌入式偏见的担忧，这些偏见可能固化刻板印象并损害公平性。此类偏见可能源于训练数据中的历史不平等、语言失衡或对抗性操纵。尽管已有缓解措施，但近期研究表明，LLMs在面对诱导偏见的对抗性攻击时仍然脆弱。本工作提出了一个可扩展的基准测试框架，用于评估LLMs对对抗性偏见诱导的鲁棒性。我们的方法包括：（i）系统性地在针对多种社会文化偏见的多个任务上探测模型，（ii）通过使用LLM-as-a-Judge方法的安全分数量化鲁棒性，以及（iii）采用越狱技术揭示安全漏洞。为促进系统性基准测试，我们发布了一个名为CLEAR-Bias的偏见相关提示词精选数据集。我们的分析将DeepSeek V3确定为最可靠的评判LLM，并揭示偏见韧性是不均匀的，其中年龄、残疾和交叉性偏见最为突出。一些小型模型在安全性上优于大型模型，这表明训练和架构可能比规模更重要。然而，没有模型对对抗性诱导完全鲁棒，使用低资源语言或拒绝抑制的越狱攻击在各类模型家族中均被证明有效。我们还发现，连续几代LLM表现出轻微的安全改进，而为医学领域微调的模型往往不如其通用对应模型安全。
英文摘要：
The growing integration of Large Language Models (LLMs) into critical societal domains has raised concerns about embedded biases that can perpetuate stereotypes and undermine fairness. Such biases may stem from historical inequalities in training data, linguistic imbalances, or adversarial manipulation. Despite mitigation efforts, recent studies show that LLMs remain vulnerable to adversarial attacks that elicit biased outputs. This work proposes a scalable benchmarking framework to assess LLM robustness to adversarial bias elicitation. Our methodology involves: (i) systematically probing models across multiple tasks targeting diverse sociocultural biases, (ii) quantifying robustness through safety scores using an LLM-as-a-Judge approach, and (iii) employing jailbreak techniques to reveal safety vulnerabilities. To facilitate systematic benchmarking, we release a curated dataset of bias-related prompts, named CLEAR-Bias. Our analysis, identifying DeepSeek V3 as the most reliable judge LLM, reveals that bias resilience is uneven, with age, disability, and intersectional biases among the most prominent. Some small models outperform larger ones in safety, suggesting that training and architecture may matter more than scale. However, no model is fully robust to adversarial elicitation, with jailbreak attacks using low-resource languages or refusal suppression proving effective across model families. We also find that successive LLM generations exhibit slight safety gains, while models fine-tuned for the medical domain tend to be less safe than their general-purpose counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
