arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.14769cs.LGcs.AI
收藏

POLCA:基于LLM的随机生成优化

POLCA: Stochastic Generative Optimization with LLM

Xuanfei Ren, Allen Nie, Tengyang Xie, Ching-An Cheng

AI总结:

本文提出POLCA框架,通过生成语言模型作为优化器,结合数值奖励和文本反馈,解决复杂系统的随机生成优化问题,实现高效探索与利用的平衡,实验表明其在多种基准测试中表现优异。

摘要:

优化复杂系统,从LLM提示到多轮代理,传统上需要大量手动迭代。我们将其形式化为随机生成优化问题,其中生成语言模型作为优化器,通过数值奖励和文本反馈发现最佳系统。我们引入优先级优化与局部上下文聚合(POLCA),一个可扩展的框架,旨在处理优化中的随机性——如噪声反馈、采样小批量和随机系统行为——同时有效管理解空间的无约束扩展。POLCA维护一个优先队列来管理探索-利用权衡,系统跟踪候选解决方案及其评估历史。为提高效率,我们整合了ε-网机制以保持参数多样性,并使用LLM总结器在历史试验中进行元学习。我们理论证明,在随机性下POLCA收敛于近最优候选解。我们在多样化的基准测试中评估了该框架,包括τ-bench、HotpotQA(代理优化)、VeriBench(代码翻译)和KernelBench(CUDA内核生成)。实验结果表明,POLCA实现了稳健、样本和时间高效的性能,在确定性和随机性问题中均优于最先进算法。该工作的代码库已公开在https://github.com/rlx-lab/POLCA。

英文摘要:

Optimizing complex systems, ranging from LLM prompts to multi-turn agents, traditionally requires labor-intensive manual iteration. We formalize this challenge as a stochastic generative optimization problem where a generative language model acts as the optimizer, guided by numerical rewards and text feedback to discover the best system. We introduce Prioritized Optimization with Local Contextual Aggregation (POLCA), a scalable framework designed to handle stochasticity in optimization -- such as noisy feedback, sampling minibatches, and stochastic system behaviors -- while effectively managing the unconstrained expansion of solution space. POLCA maintains a priority queue to manage the exploration-exploitation tradeoff, systematically tracking candidate solutions and their evaluation histories. To enhance efficiency, we integrate an $\varepsilon$-Net mechanism to maintain parameter diversity and an LLM Summarizer to perform meta-learning across historical trials. We theoretically prove that POLCA converges to near-optimal candidate solutions under stochasticity. We evaluate our framework on diverse benchmarks, including $τ$-bench, HotpotQA (agent optimization), VeriBench (code translation) and KernelBench (CUDA kernel generation). Experimental results demonstrate that POLCA achieves robust, sample and time-efficient performance, consistently outperforming state-of-the-art algorithms in both deterministic and stochastic problems. The codebase for this work is publicly available at https://github.com/rlx-lab/POLCA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑