SpecEval：评估模型对行为规范的遵循程度
SpecEval: Evaluating Model Adherence to Behavior Specifications
AI总结：
本文提出SpecEval自动化框架，通过解析行为规范、生成提示并利用模型评判，审计基础模型对提供商规范的遵循程度，发现跨提供商存在高达20%的遵循缺口。
摘要：
开发基础模型的公司会发布其承诺模型将遵循的行为指南，但模型是否确实遵循这些指南仍不明确。尽管OpenAI、Anthropic和Google等提供商已发布详细规范，描述了其模型所需的安全约束和定性特征，但尚未对这些指南的遵循情况进行系统性审计。我们引入了一个自动化框架，通过解析行为陈述、生成针对性提示，并使用模型来评判遵循程度，从而根据提供商的规范对模型进行审计。我们的核心关注点是提供商规范、其模型输出及其自身模型作为评判者之间的三方一致性；这是先前两方生成器-验证器一致性的扩展。这建立了一个必要的基线：至少，基础模型在由开发者的评估模型评判时，应始终满足开发者的行为规范。我们将该框架应用于来自六家开发商的16个模型，涉及超过100条行为陈述，发现了系统性的不一致性，包括各提供商之间高达20%的遵循缺口。
英文摘要：
Companies that develop foundation models publish behavioral guidelines they pledge their models will follow, but it remains unclear if models actually do so. While providers such as OpenAI, Anthropic, and Google have published detailed specifications describing both desired safety constraints and qualitative traits for their models, there has been no systematic audit of adherence to these guidelines. We introduce an automated framework that audits models against their providers specifications by parsing behavioral statements, generating targeted prompts, and using models to judge adherence. Our central focus is on three way consistency between a provider specification, its model outputs, and its own models as judges; an extension of prior two way generator validator consistency. This establishes a necessary baseline: at minimum, a foundation model should consistently satisfy the developer behavioral specifications when judged by the developer evaluator models. We apply our framework to 16 models from six developers across more than 100 behavioral statements, finding systematic inconsistencies including compliance gaps of up to 20 percent across providers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
