arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03358cs.AIcs.CLcs.LG
收藏

GFlowPO：生成流网络作为语言模型提示优化器

GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer

Korea Advanced Institute of Science；Korea University；Mila -- Qu\'ebec AI Institute

Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

GFlowPO通过生成流网络和动态内存更新机制，提升语言模型提示优化的样本效率和性能。

摘要：

在语言模型（LMs）中寻找有效的提示至关重要且 notoriously 困难：提示空间是组合性巨大的，由于昂贵的目标LM评估，奖励稀疏。然而，现有的基于强化学习的提示优化器往往依赖于 on-policy 更新和一个从固定分布中采样的元提示，导致样本效率低下。我们提出了GFlowPO，一个概率提示优化框架，将提示搜索视为一个后验推断问题，通过一个由元提示引导的参考LM先验进行正则化。第一步，我们使用基于生成流网络（GFlowNet）的目标对轻量级提示LM进行微调，使用基于回放的训练策略，重用过去的提示评估以实现样本高效的探索。第二步，我们引入动态内存更新（DMU），一种无需训练的机制，通过注入（i）来自回放缓冲区的多样化提示和（ii）来自小优先队列的高性能提示，从而逐步将搜索过程集中在高奖励区域。在少样本文本分类、指令诱导基准和问答任务中，GFlowPO始终优于最近的离散提示优化基线。

英文摘要：

Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑