外向还是内向？控制大语言模型的性格
Extroversion or Introversion? Controlling The Personality of Your Large Language Models
AI总结：
本文系统研究了大语言模型性格控制方法，发现提示、SFT、RLHF和持续预训练的有效性依次递减，并提出结合SFT与提示的PISF策略，实现高效且稳健的性格控制。
摘要：
大语言模型（LLMs）在文本生成和理解方面展现出强大的能力，能够模仿人类行为并表现出合成性格。然而，一些大语言模型表现出冒犯性性格，传播有害言论。现有文献忽视了大语言模型性格的起源与演变，以及有效的性格控制方法。为填补这些空白，本研究对大语言模型性格控制进行了全面调查。我们考察了若干种影响大语言模型的典型方法，包括三种训练方法：持续预训练（Continual Pre-training）、监督微调（SFT）和基于人类反馈的强化学习（RLHF），以及推理阶段的考量（提示）。调查揭示了控制有效性的层级关系：提示 > SFT > RLHF > 持续预训练。值得注意的是，与提示诱导相比，SFT表现出更高的控制成功率。虽然提示非常有效，但我们发现提示诱导出的性格不如训练得到的性格稳健，使其在反向性格提示诱导下更容易表现出相互矛盾的性格。此外，结合SFT和提示的优势，我们提出了监督微调后提示诱导（PISF）方法，该方法成为控制大语言模型性格最有效且最稳健的策略，展现出高效性、高成功率和强稳健性。即使在反向性格提示诱导下，由PISF控制的大语言模型仍表现出稳定且稳健的性格。
英文摘要：
Large language models (LLMs) exhibit robust capabilities in text generation and comprehension, mimicking human behavior and exhibiting synthetic personalities. However, some LLMs have displayed offensive personality, propagating toxic discourse. Existing literature neglects the origin and evolution of LLM personalities, as well as the effective personality control. To fill these gaps, our study embarked on a comprehensive investigation into LLM personality control. We investigated several typical methods to influence LLMs, including three training methods: Continual Pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF), along with inference phase considerations (prompts). Our investigation revealed a hierarchy of effectiveness in control: Prompt > SFT > RLHF > Continual Pre-train. Notably, SFT exhibits a higher control success rate compared to prompt induction. While prompts prove highly effective, we found that prompt-induced personalities are less robust than those trained, making them more prone to showing conflicting personalities under reverse personality prompt induction. Besides, harnessing the strengths of both SFT and prompt, we proposed $\underline{\text{P}}$rompt $\underline{\text{I}}$nduction post $\underline{\text{S}}$upervised $\underline{\text{F}}$ine-tuning (PISF), which emerges as the most effective and robust strategy for controlling LLMs' personality, displaying high efficacy, high success rates, and high robustness. Even under reverse personality prompt induction, LLMs controlled by PISF still exhibit stable and robust personalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
