上下文演示的重要性：面向伪监督精炼的提示优化
In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement
AI总结：
针对LLM微调中伪监督易过拟合的问题，提出PAPO算法，联合优化提示与伪监督，利用梯度转文本批评迭代精炼，在问答、NLI和分子优化任务上验证了有效性。
摘要：
大型语言模型（LLMs）在各种任务中取得了巨大成功，有时需要微调以进一步提升生成质量。现有大多数方法依赖人工监督或参数重训练，这两者在数据收集和计算资源方面成本高昂。为应对这些挑战，一个直接解决方案是从无监督下游任务中生成“高置信度”数据，并将其用于上下文提示或提示优化，以精炼伪监督。然而，仅依赖此类数据可能导致过拟合。本文利用LLMs的上下文学习（ICL）能力，提出一种新方法——伪监督演示对齐提示优化（PAPO）算法，该算法联合精炼提示和整体伪监督。所提出的学习目标确保优化后的提示在将下游任务的伪监督数据用作演示时，引导LLM对给定输入生成一致的响应，从而实现对整体伪监督的精炼。提示通过将梯度信号转化为文本批评进行优化，这些批评作为反馈迭代精炼提示和模型响应。在简化分类设置中的理论分析表明，精炼后的伪监督展现出几何聚类结构，有助于缓解过拟合。在问答、自然语言推理基准以及一个真实世界的分子优化任务上的实验，证明了所提算法的有效性。
英文摘要：
Large language models (LLMs) have achieved great success across diverse tasks, and fine-tuning is sometimes needed to further enhance generation quality. Most existing methods rely on human supervision or parameter retraining, both of which are costly in terms of data collection and computational resources. To handle these challenges, a direct solution is to generate ``high-confidence'' data from unsupervised downstream tasks and use them for in-context prompting or prompt optimization to refine the pseudo-supervision. However, relying solely on such data may lead to overfitting. In this paper, we leverage the in-context learning (ICL) abilities of LLMs and propose a novel approach, pseudo-supervised demonstrations aligned prompt optimization (PAPO) algorithm, which jointly refines both the prompt and the overall pseudo-supervision. The proposed learning objective ensures that the optimized prompt guides the LLM to generate consistent responses for a given input when pseudo-supervised data from the downstream task are used as demonstrations, enabling refinement over the entire pseudo-supervision. The prompt is optimized by translating gradient signals into textual critiques, which serve as feedback to iteratively refine the prompt and model responses. Theoretical analysis in a simplified classification setting shows that the refined pseudo-supervision exhibits a geometric clustering structure, helping to mitigate overfitting. Experiments on question answering, natural language inference benchmarks, and a real-world molecule optimization task, show the effectiveness of the proposed algorithm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
