arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09741cs.LGcs.AI
收藏

ExecTune: 通过引导模型有效控制黑盒大语言模型

ExecTune: Effective Steering of Black-Box LLMs with Guide Models

AWS AI(亚马逊云科技人工智能)

Vijay Lingam, Aditya Golatkar, Anwesan Pal, Ben Vo, Narayanan Sadagopan, Alessandro Achille, Jun Huan, Anoop Deoras, Stefano Soatto

AI总结:

本文提出ExecTune方法,通过结合教师引导采样、监督微调和结构感知强化学习,优化引导模型生成策略的可执行性与效率,提升大语言模型在数学推理和代码生成任务中的性能。

摘要:

对于通过黑盒API部署的大语言模型,反复的推理成本常超过一次训练成本。这促使了复合代理系统,将昂贵的推理成本摊销到可重用的中间表示中。我们研究了一类广泛的系统,称为引导-核心策略(GCoP),其中引导模型生成结构化策略,由黑盒核心模型执行。这种抽象涵盖了基础、监督和顾问式方法,主要区别在于引导模型的训练方式。我们以成本敏感的效用目标正式化GCoP,并证明端到端性能由引导平均可执行性决定:引导生成的策略被核心模型忠实执行的概率。我们的分析显示,现有GCoP实例在部署约束下往往未能优化可执行性,导致策略脆弱和计算效率低下。受此启发,我们提出ExecTune,一种结合教师引导采样、监督微调和结构感知强化学习的训练方法,直接优化语法有效性、执行成功率和成本效率。在数学推理和代码生成基准上,GCoP结合ExecTune在准确性上比现有最先进基线提高9.2%,同时将推理成本降低22.4%。它使Claude Haiku 3.5在数学和代码任务中超越Sonnet 3.5,并在38%更低的成本下接近Sonnet 4的1.7%绝对准确度。此外,GCoP还支持模块化适应,通过更新引导模型而不重新训练核心模型。

英文摘要:

For large language models deployed through black-box APIs, recurring inference costs often exceed one-time training costs. This motivates composed agentic systems that amortize expensive reasoning into reusable intermediate representations. We study a broad class of such systems, termed Guide-Core Policies (GCoP), in which a guide model generates a structured strategy that is executed by a black-box core model. This abstraction subsumes base, supervised, and advisor-style approaches, which differ primarily in how the guide is trained. We formalize GCoP under a cost-sensitive utility objective and show that end-to-end performance is governed by guide-averaged executability: the probability that a strategy generated by the guide can be faithfully executed by the core. Our analysis shows that existing GCoP instantiations often fail to optimize executability under deployment constraints, resulting in brittle strategies and inefficient computation. Motivated by these insights, we propose ExecTune, a principled training recipe that combines teacher-guided acceptance sampling, supervised fine-tuning, and structure-aware reinforcement learning to directly optimize syntactic validity, execution success, and cost efficiency. Across mathematical reasoning and code-generation benchmarks, GCoP with ExecTune improves accuracy by up to 9.2% over prior state-of-the-art baselines while reducing inference cost by up to 22.4%. It enables Claude Haiku 3.5 to outperform Sonnet 3.5 on both math and code tasks, and to come within 1.7% absolute accuracy of Sonnet 4 at 38% lower cost. Beyond efficiency, GCoP also supports modular adaptation by updating the guide without retraining the core.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑