揭秘语言模型中的优化提示词
Demystifying optimized prompts in language models
AI总结：
该研究针对语言模型中可调控输出却无法解释的机器生成优化提示词，探究其构成与模型解析机制，发现其多含罕见名词与标点，可通过稀疏激活区分，且在不同指令微调模型中表征形成路径相似。
摘要：
现代语言模型（LM）对分布外输入的鲁棒性不足。机器生成的（“优化的”）提示词可用于调节LM的输出并诱发特定行为，同时看起来完全无法解释。在本研究中，我们探究了优化提示词的构成，以及LM解析优化提示词并据此生成预测的机制。我们发现，优化提示词主要由标点符号和在训练数据中更为罕见的名词词元构成。在内部机制层面，基于模型激活的稀疏子集，可清晰区分优化提示词与自然语言提示词。在不同系列的指令微调模型中，优化提示词的表征在网络中形成的路径具有相似性。
英文摘要：
Modern language models (LMs) are not robust to out-of-distribution inputs. Machine generated (``optimized'') prompts can be used to modulate LM outputs and induce specific behaviors while appearing completely uninterpretable. In this work, we investigate the composition of optimized prompts, as well as the mechanisms by which LMs parse and build predictions from optimized prompts. We find that optimized prompts primarily consist of punctuation and noun tokens which are more rare in the training data. Internally, optimized prompts are clearly distinguishable from natural language counterparts based on sparse subsets of the model's activations. Across various families of instruction-tuned models, optimized prompts follow a similar path in how their representations form through the network.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
