多步推理和工具使用中黑盒LLM的提示策略：基于经验迭代蒸馏的强化学习框架
Prompting Policies for Multi-step Reasoning and Tool-Use in Black-box LLMs with Iterative Distillation of Experience
AI总结：
本文提出基于经验迭代蒸馏的强化学习框架，用于训练提示策略以提升黑盒LLM的多步推理和工具使用能力，实验显示在逻辑推理和工具使用任务中性能显著提升。
摘要：
向冻结的'黑盒'大型语言模型（LLM）交互的趋势将提示工程从启发式练习转变为关键优化挑战。我们提出了一种强化学习（RL）框架，通过经验迭代蒸馏来训练学习提示策略。在该架构中，一个轻量级提示器模型被优化以最大化任务特定的奖励，用于更大的冻结工人LLM。通过利用将标量奖励与密集文本批评相结合的对比经验缓冲区，我们的方法有效将迭代提示细化 amortizes 到单次策略权重中。我们的实验分析聚焦于Big Bench Extra Hard（BBEH）和Tau-bench套件，涵盖多样化的多步推理和工具使用任务。我们展示了显著的收益，逻辑密集推理任务性能从55%提升到90%，工具使用任务性能从74%提升到91%。此外，我们分析了提示的结构演变，展示了策略如何发现专门的算法启发式方法。我们提供了与最先进的进化基线如GEPA的全面比较，表明迭代蒸馏在更高的样本效率下实现了更优的性能。
英文摘要：
The shift toward interacting with frozen, "black-box" Large Language Models (LLMs) has transformed prompt engineering from a heuristic exercise into a critical optimization challenge. We propose a Reinforcement Learning (RL) framework for training learned prompting policies via iterative distillation of experience. In this architecture, a lightweight prompter model is optimized to maximize task-specific rewards for a larger, frozen worker LLM. By utilizing a contrastive experience buffer that couples scalar rewards with dense textual critiques, our approach effectively amortizes iterative prompt refinement into single-shot policy weights. Our experimental analysis focuses on the Big Bench Extra Hard (BBEH) and Tau-bench suites, covering a diverse range of multi-step reasoning and tool-use tasks. We demonstrate significant gains, improving performance from 55% to 90% in logic-intensive reasoning and 74% to 91% in tool-use tasks. Furthermore, we analyze the structural evolution of prompts, demonstrating how the policy discovers specialized algorithmic heuristics. We provide comprehensive comparisons against state-of-the-art evolutionary baselines like GEPA, showing that iterative distillation achieves superior performance with higher sample efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
