自指令派生提示生成遇上上下文学习：解锁黑盒大语言模型的新潜力
Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs
AI总结：
本文提出自指令上下文学习框架，通过强化学习生成可靠派生提示并构建上下文演示，从而提升黑盒大语言模型如GPT-4的响应质量。
摘要：
大语言模型（LLMs）已在生成高质量响应方面展现出成功。为了实现与人类偏好更好的对齐，各种工作基于特定的优化过程被提出，然而这些方法并不适用于像GPT-4这样的黑盒大语言模型，因为其参数不可访问。在黑盒大语言模型的情况下，其性能高度依赖于所提供提示的质量。现有的提升响应质量的方法通常涉及提示精炼模型，但这些方法可能面临精炼提示与原始提示之间语义不一致的问题，并且通常忽视二者之间的关系。为应对这些挑战，我们引入了一个自指令上下文学习框架，通过生成可靠的派生提示来构建信息丰富的上下文环境，从而使大语言模型能够给出更有效的响应。我们的方法结合了自指令强化学习机制，在派生提示生成过程中能够与响应模型直接交互，以实现更好的对齐。随后，我们将查询形式化为一个上下文学习任务，利用大语言模型结合派生提示生成的响应，为原始提示构建上下文演示。该策略确保与原始查询保持一致，减少与精炼提示之间的偏差，并最大化大语言模型的上下文学习能力。大量实验表明，所提出的方法不仅能生成更可靠的派生提示，还能显著增强大语言模型给出更有效响应的能力，包括GPT-4等黑盒模型。
英文摘要：
Large language models (LLMs) have shown success in generating high-quality responses. In order to achieve better alignment with LLMs with human preference, various works are proposed based on specific optimization process, which, however, is not suitable to Black-Box LLMs like GPT-4, due to inaccessible parameters. In Black-Box LLMs case, their performance is highly dependent on the quality of the provided prompts. Existing methods to enhance response quality often involve a prompt refinement model, yet these approaches potentially suffer from semantic inconsistencies between the refined and original prompts, and typically overlook the relationship between them. To address these challenges, we introduce a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments. Our approach incorporates a self-instructed reinforcement learning mechanism, enabling direct interaction with the response model during derived prompt generation for better alignment. We then formulate querying as an in-context learning task, using responses from LLMs combined with the derived prompts to establish a contextual demonstration for the original prompt. This strategy ensures alignment with the original query, reduces discrepancies from refined prompts, and maximizes the LLMs' in-context learning capability. Extensive experiments demonstrate that the proposed method not only generates more reliable derived prompts but also significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
