arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.20657cs.CL
收藏

DORY:面向大语言模型的审慎式提示恢复方法

DORY: Deliberative Prompt Recovery for LLM

Zhejiang University(浙江大学)

Lirong Gao, Ru Peng, Yiming Zhang, Junbo Zhao

AI总结:

本文提出DORY方法,利用输出概率不确定性与提示恢复成功率之间的负相关性,通过草稿重建、提示精炼和噪声过滤,在仅使用单个LLM的条件下将提示恢复性能提升约10.82%,达到新的最优水平。

摘要:

大语言模型(LLMs)中的提示恢复对于理解LLM的工作机制以及应对隐私、版权等问题至关重要。仅提供推理接口的趋势限制了获取恢复所需关键输出的途径,从而使该任务更加困难。为应对这一挑战,我们从有限输出中提取与提示相关的信息,并发现基于输出概率的不确定性与提示恢复成功率之间存在强(负)相关关系。基于这一发现,我们提出了审慎式提示恢复方法DORY(Deliberative PrOmpt RecoverY),利用不确定性来准确恢复提示。DORY包括从输出中重建草稿、借助提示线索进行精炼,以及基于不确定性过滤噪声。我们在多种LLM和提示基准上的评估表明,DORY优于现有基线方法,性能提升约10.82%,并在提示恢复任务中创造了新的最优水平。值得注意的是,DORY仅使用单个LLM,无需任何外部资源或模型,提供了一种成本效益高且用户友好的提示恢复解决方案。

英文摘要:

Prompt recovery in large language models (LLMs) is crucial for understanding how LLMs work and addressing concerns regarding privacy, copyright, etc. The trend towards inference-only APIs complicates this task by restricting access to essential outputs for recovery. To tackle this challenge, we extract prompt-related information from limited outputs and identify a strong(negative) correlation between output probability-based uncertainty and the success of prompt recovery. This finding led to the development of Deliberative PrOmpt RecoverY (DORY), our novel approach that leverages uncertainty to recover prompts accurately. DORY involves reconstructing drafts from outputs, refining these with hints, and filtering out noise based on uncertainty. Our evaluation across diverse LLMs and prompt benchmarks shows that DORY outperforms existing baselines, improving performance by approximately 10.82% and establishing a new state-of-the-art record in prompt recovery tasks. Significantly, DORY operates using a single LLM without any external resources or model, offering a cost-effective, user-friendly prompt recovery solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑