衡量与基准测试大语言模型生成说服性语言的能力
Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language
AI总结：
该研究构建Persuasive-Pairs数据集并训练回归评分模型，跨领域基准测试大语言模型在说服性改写和释义场景下的说服性语言生成能力，还发现LLaMA3的角色设定会显著影响其说服性语言输出。
摘要：
我们身处大量试图影响我们的信息中，比如预告消息、辩论、带有政治框架的新闻以及宣传内容——所有这些都使用了说服性语言。随着近期大语言模型（LLMs）受到广泛关注，我们研究了LLMs生成说服性文本的能力。与以往聚焦于特定领域或说服类型的研究不同，我们开展了一项跨多个领域的通用研究，以衡量并基准测试LLMs生成说服性语言的程度——既包括明确指示其改写文本以增强或减弱说服性的场景，也包括仅指示其进行释义的场景。我们构建了新数据集Persuasive-Pairs，该数据集由短文本与LLM为增强或削弱说服性语言而生成的改写文本组成的文本对构成。我们对这些文本对的说服性语言进行了相对尺度的多标注：这本身就是一份有价值的资源，同时也可用于训练一个回归模型，对说服性语言进行评分和基准测试，包括对跨领域的新型LLM进行评估。在我们的分析中发现，LLaMA3系统提示中的不同“角色设定”会大幅改变说服性语言的生成，即便仅指示模型进行释义时也是如此。
英文摘要：
We are exposed to much information trying to influence us, such as teaser messages, debates, politically framed news, and propaganda - all of which use persuasive language. With the recent interest in Large Language Models (LLMs), we study the ability of LLMs to produce persuasive text. As opposed to prior work which focuses on particular domains or types of persuasion, we conduct a general study across various domains to measure and benchmark to what degree LLMs produce persuasive language - both when explicitly instructed to rewrite text to be more or less persuasive and when only instructed to paraphrase. We construct the new dataset Persuasive-Pairs of pairs of a short text and its rewrite by an LLM to amplify or diminish persuasive language. We multi-annotate the pairs on a relative scale for persuasive language: a valuable resource in itself, and for training a regression model to score and benchmark persuasive language, including for new LLMs across domains. In our analysis, we find that different 'personas' in LLaMA3's system prompt change persuasive language substantially, even when only instructed to paraphrase.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
