发表机构
SURUS（SURUS）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Benchy提出一种语义语言与执行引擎，通过规范YAML编译为JSON中间表示，统一基准测试定义与执行，实现AI系统与基准测试的分离及通用运行时契约。
AI 中文摘要
Benchy是一种用于基准测试AI程序的语义语言和执行引擎。一个基准测试完全由程序、评分函数和数据集指定，即B=(P,S,D)，并且与接受该测试的AI系统分离；一次运行将两者绑定，即R=(B,AI)。基准测试以规范的YAML格式编写，其中每个语义概念只有一种有效语法，通过共享的任务/领域/语言本体进行分类，并确定性地编译为引擎执行的规范JSON中间表示。编译改变表示形式而非含义：它不修复无效定义，也不注入隐藏默认值。程序使用固定的命名输入和输出字段模式，叶级输出字段即评分维度，引擎暴露一个通用运行时契约——输入为命名字段对象，输出为命名字段对象——外部AI系统在边界处适配该契约，因此集成机制永远不会传播到基准测试语义中。本文给出了语义对象模型、本体和任务到程序的验证规则、评分和失败语义、编译和执行架构，以及当前语言的范围。附录为第一个引擎实现规定了规范性工程契约。
英文摘要
Benchy is a semantic language and execution engine for benchmarking AI programs. A benchmark is completely specified by a program, a scoring function, and a dataset, B=(P,S,D), and is separate from the AI-system taking it; a run binds the two, R=(B,AI). Benchmarks are authored as canonical YAML in which each semantic concept has one valid syntax, classified by a shared task/domain/language ontology, and deterministically compiled into a canonical JSON intermediate representation that the engine executes. Compilation changes representation, not meaning: it does not repair invalid definitions or inject hidden defaults. Programs use fixed schemas of named input and output fields, the leaf output fields are the scoring dimensions, and the engine exposes one universal runtime contract --- a named-field input object in, a named-field output object out --- to which external AI-systems adapt at the boundary, so integration mechanics never propagate into benchmark semantics. This paper gives the semantic object model, the ontology and task-to-program validation rule, the scoring and failure semantics, the compilation and execution architecture, and the scope of the current language. An appendix fixes the normative engineering contract for the first engine implementation.
Comments21 pages
