arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.08348cs.CL
收藏

即时任性:连续事件离散化解释的奇怪案例

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

University of Washington(华盛顿大学);Allen Institute for AI(艾伦人工智能研究所);University of California-Irvine(加州大学欧文分校)

Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi

摘要:

微调目标任务的连续提示最近出现了一个紧凑的替代全模型微调。出于这些有希望的结果,我们研究的可行性提取离散(文本)的解释连续提示,是忠实于他们解决的问题。在实践中,我们观察到连续提示解决的任务与其最近邻离散投影之间的“任性”行为:我们可以找到解决任务的连续提示,同时被投影到任意文本(例如,不同或甚至矛盾的任务的定义),同时在用于该任务的相同大小的最佳连续提示的非常小(2%)的裕度内。我们提供了这种奇怪和令人惊讶的行为背后的直觉,以及量化各种参数的影响的广泛的实证分析。例如,对于更大的模型大小,我们观察到更高的任性,也就是说,我们可以找到更紧密地映射到任何任意文本的提示,并且准确性下降较小。这些研究结果具有重要的意义,涉及到忠实地解释连续提示的难度及其在模型和任务中的泛化,为提示语言模型的未来进展提供指导。

英文摘要:

Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a "wayward" behavior between the task solved by continuous prompts and their nearest neighbor discrete projections: We can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task), while being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of various parameters. For instance, for larger model sizes we observe higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop in accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑