aiXamine：简化的大语言模型安全与安保评估
aiXamine: Simplified LLM Safety and Security
AI总结：
aiXamine是一个黑盒评估平台，整合40多项测试和八项服务，全面评估LLM安全与安保，发现领先模型存在对抗攻击、偏见和隐私漏洞，并揭示开源模型在特定维度可匹敌专有模型。
摘要：
评估大语言模型（LLM）的安全与安保仍然是一项复杂任务，通常要求用户穿越由临时基准、数据集、指标和报告格式构成的碎片化景观。为应对这一挑战，我们提出了aiXamine，一个针对LLM安全与安保的全面黑盒评估平台。aiXamine整合了超过40项测试（即基准），这些测试被组织成八项关键服务，针对安全与安保的特定维度：对抗鲁棒性、代码安全、公平性与偏见、幻觉、模型与数据隐私、分布外（OOD）鲁棒性、过度拒答以及安全对齐。该平台将评估结果汇总为每个模型的单一详细报告，提供模型性能的详细分解、测试示例和丰富的可视化。我们使用aiXamine评估了超过50个公开可获取和专有的LLM，进行了超过2000次检查。我们的发现揭示了领先模型中的显著漏洞，包括OpenAI的GPT-4o易受对抗性攻击、xAI的Grok-3存在有偏输出，以及Google的Gemini 2.0存在隐私弱点。此外，我们观察到开源模型在特定服务（如安全对齐、公平性与偏见以及OOD鲁棒性）中可以匹敌甚至超越专有模型。最后，我们识别了蒸馏策略、模型规模、训练方法和架构选择之间的权衡。
英文摘要：
Evaluating Large Language Models (LLMs) for safety and security remains a complex task, often requiring users to navigate a fragmented landscape of ad hoc benchmarks, datasets, metrics, and reporting formats. To address this challenge, we present aiXamine, a comprehensive black-box evaluation platform for LLM safety and security. aiXamine integrates over 40 tests (i.e., benchmarks) organized into eight key services targeting specific dimensions of safety and security: adversarial robustness, code security, fairness and bias, hallucination, model and data privacy, out-of-distribution (OOD) robustness, over-refusal, and safety alignment. The platform aggregates the evaluation results into a single detailed report per model, providing a detailed breakdown of model performance, test examples, and rich visualizations. We used aiXamine to assess over 50 publicly available and proprietary LLMs, conducting over 2K examinations. Our findings reveal notable vulnerabilities in leading models, including susceptibility to adversarial attacks in OpenAI's GPT-4o, biased outputs in xAI's Grok-3, and privacy weaknesses in Google's Gemini 2.0. Additionally, we observe that open-source models can match or exceed proprietary models in specific services such as safety alignment, fairness and bias, and OOD robustness. Finally, we identify trade-offs between distillation strategies, model size, training methods, and architectural choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
