当“有用的助手”并非真的有用：系统提示中的角色设定并不会提升大语言模型的性能
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
AI总结：
该研究系统评估系统提示中的角色设定对LLM性能的影响，发现添加角色无法普遍提升事实性任务表现，最优角色自动识别难度大，角色效果多具随机性。
摘要：
提示工程是人类与大语言模型（LLM）交互的主要方式。商业AI系统通常会在系统提示中定义LLM的角色，例如ChatGPT将“你是一个有用的助手”作为其默认系统提示的一部分。尽管目前业界普遍会在系统提示中加入角色设定，但不同角色如何影响模型在客观任务上的性能仍不明确。本研究对系统提示中的角色设定开展了系统性评估。我们整理了一份包含162个角色的列表，涵盖6类人际关系和8个专业领域。通过对4个主流LLM模型家族以及2410个事实性问题的广泛分析，我们证明：与未添加角色的对照组设置相比，在系统提示中加入角色并不能提升模型在各类问题上的性能。尽管如此，进一步分析表明，角色的性别、类型和所属领域都会影响最终的预测准确率。我们还测试了一系列角色搜索策略，结果发现，虽然为每个问题聚合最优角色对应的结果能显著提升预测准确率，但自动识别最优角色的难度很大，其预测表现通常并不优于随机选择。总体而言，我们的研究结果表明，尽管添加角色在某些场景下可能带来性能提升，但每个角色的效果在很大程度上具有随机性。代码和数据可在https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles获取。
英文摘要：
Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses ``You are a helpful assistant'' as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model's performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
