面向LLM推理的定向测试:一种受审计约束的协议
Targeted Tests for LLM Reasoning: An Audit-Constrained Protocol
AI总结:
本文提出一种受审计约束的协议,用于评估LLM推理能力,通过组件自适应提示采样与均匀采样对比,验证了在受控环境下研究定向提示变化的有效性。
摘要:
固定推理基准评估标准提示,但语义上有效的呈现变化仍可能改变模型行为。提示变化研究可揭示此类失败,但缺乏审计时可能混杂真实模型错误、无效扰动、提取伪影和不匹配的搜索过程。本文提出一种受审计约束的定向推理评估协议。提示变体由有限组件语法生成,确定性渲染,固定查询预算下评估,并在经过语义和提取审计后才视为模型错误。在此协议中,我们实例化了组件自适应提示采样(CAPS),一种基于得分的提示组件采样器,并在相同任务库、渲染器、模型接口、解码设置和审计程序下,与等预算均匀组件采样进行比较。在三个受审计的切片中,该协议确认了模型错误提示键,同时排除了格式和提取伪影,但匹配比较未显示CAPS在受控产量或唯一提示键发现上优于均匀采样。贡献是方法论的:定向提示变化可以在可重建、可审查、预算匹配的协议下研究,代理引导策略应通过受控产量而非原始不匹配计数或选定示例单独判断。
英文摘要:
Fixed reasoning benchmarks evaluate canonical prompts, but semantically valid changes in presentation can still change model behavior. Studies of prompt variation can reveal such failures, but without audit they can mix genuine model errors with invalid perturbations, extraction artifacts, and unmatched search procedures. We propose an audit-constrained protocol for targeted reasoning evaluation. Prompt variants are generated from a finite component grammar, rendered deterministically, evaluated under a fixed query budget, and counted as model errors only after semantic and extraction audit. Within this protocol we instantiate Component-Adaptive Prompt Sampling (CAPS), a score-based sampler over prompt components, and compare it with equal-budget uniform component sampling under the same task bank, renderer, model interface, decoding settings, and audit procedure. Across three audited slices, the protocol identifies confirmed model-error prompt keys while excluding formatting and extraction artifacts, but matched comparisons do not show that CAPS improves audited yield or unique prompt-key discovery over uniform sampling. The contribution is methodological: targeted prompt variation can be studied under a reconstructable, reviewable, budget-matched protocol, and proxy-guided policies should be judged by audited yield rather than raw mismatch counts or selected examples alone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。