arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.12947stat.MLcs.AIcs.LGstat.ME
收藏

何时应释放AI工作流?始终有效的推理用于黑盒生成-验证系统

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems

Department of Statistics, Purdue University(普渡大学统计系)

Young Hyun Cho, Will Wei Sun

AI总结:

本文提出一种始终有效的释放包装器,用于现有生成-验证流程,通过构建高质量失败参考池并校准部署时的验证器评分,以在可选停止条件下保证有效性。

摘要:

LLM赋能的AI工作流越来越多地通过迭代的生成-评估-修订循环产生输出。每次迭代都能改进候选方案,但也会产生一个释放决策:何时停止并输出当前结果?这提出了统计挑战,因为部署时的评估器评分是自适应生成并反复监控的,但通常用于校准的似然模型或交换性假设不可用。我们提出了一种始终有效的释放包装器,用于现有生成-验证流程。该包装器构建了一个高质量失败参考池,将部署时的评估器评分与该池校准,并通过e过程积累所得证据。这分离了两种角色:参考池将黑盒评分转化为保守证据,而e过程在可选停止条件下提供有效性。在理论方面,我们证明保守的参考池能保证有限样本下在不可行任务上的释放概率受控,即那些给定工作流无法产生可靠解决方案的任务。我们还刻画了在何种条件下,相同的保守规则仍能在可行任务上实现非平凡的释放。在MBPP+编码代理案例研究中,该包装器减少了相对于基线停止规则的过早错误释放,同时仍能释放那些工作流反复积累中等支持证据的任务。

英文摘要:

LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but it also creates a release decision: when to stop and output the current result? This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored, yet the likelihood models or exchangeability assumptions typically used for calibration are unavailable. We propose an always-valid release wrapper for existing generator-evaluator pipelines. The wrapper builds a hard-negative reference pool of high-scoring failures, calibrates deployment-time evaluator scores against this pool, and accumulates the resulting evidence with an e-process. This separates two roles: the reference pool turns black-box scores into conservative evidence, while the e-process provides validity under optional stopping. In theory, we show that a conservative reference pool yields finite-sample control of the probability of releasing on infeasible tasks, that is, tasks for which the given workflow is not capable of producing a reliable solution. We also characterize conditions under which the same conservative rule still achieves nontrivial release on feasible tasks. In an MBPP+ coding-agent case study, the wrapper reduces premature incorrect release relative to baseline stopping rules while still releasing on tasks for which the workflow repeatedly accumulates moderate supporting evidence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑