衡量并提升大语言模型的说服力
Measuring and Improving Persuasiveness of Large Language Models
AI总结：
该研究推出首个大规模说服力基准与竞技场平台PersuasionBench、PersuasionArena，自动衡量生成式模型说服能力，发现LLM说服力与规模正相关但小模型经针对性训练可超越大模型，对AI监管的规模依赖假设提出挑战。
摘要：
大语言模型（LLM）正越来越多地被用于涉及生成供人类消费内容（例如营销内容）的工作流中，也被用于直接与人类互动的场景（例如通过聊天机器人）。开发这类能够生成经证实具备说服力的信息的系统，对社会而言既有机遇也有挑战。一方面，这类系统可以对广告、社会公益（例如解决药物成瘾问题）等领域产生积极影响；另一方面，它们也可能被滥用于传播错误信息、塑造政治观点。为引导LLM对社会的影响，我们需要开发系统来衡量并基准测试其说服力。基于这一动机，我们推出了PersuasionBench（说服力基准）和PersuasionArena（说服力竞技场），这是首个包含一系列任务的大规模基准与竞技场平台，可自动衡量生成式模型的说服能力。我们研究了LLM对有助于生成更具说服力语言的语言模式的了解与利用程度。研究结果表明，LLM的说服力与模型规模呈正相关，但较小的模型也可以被打造得比大得多的模型具备更强的说服力。值得注意的是，使用合成数据集与自然数据集进行针对性训练，能显著提升较小模型的说服能力，这对依赖规模的假设提出了挑战。我们的研究结果对模型开发者与政策制定者都有关键启示。例如，尽管《欧盟人工智能法案》与加州SB-1047法案旨在基于浮点运算次数监管AI模型，但我们证明，仅靠这类简单指标无法涵盖AI社会影响的全部范围。我们邀请学界探索并为PersuasionArena和PersuasionBench贡献力量，相关资源可访问https://bit.ly/measure-persuasion，以推动我们对AI驱动的说服行为及其社会影响的理解。
英文摘要：
LLMs are increasingly being used in workflows involving generating content to be consumed by humans (e.g., marketing) and also in directly interacting with humans (e.g., through chatbots). The development of such systems that are capable of generating verifiably persuasive messages presents both opportunities and challenges for society. On the one hand, such systems could positively impact domains like advertising and social good, such as addressing drug addiction, and on the other, they could be misused for spreading misinformation and shaping political opinions. To channel LLMs' impact on society, we need to develop systems to measure and benchmark their persuasiveness. With this motivation, we introduce PersuasionBench and PersuasionArena, the first large-scale benchmark and arena containing a battery of tasks to measure the persuasion ability of generative models automatically. We investigate to what extent LLMs know and leverage linguistic patterns that can help them generate more persuasive language. Our findings indicate that the persuasiveness of LLMs correlates positively with model size, but smaller models can also be made to have a higher persuasiveness than much larger models. Notably, targeted training using synthetic and natural datasets significantly enhances smaller models' persuasive capabilities, challenging scale-dependent assumptions. Our findings carry key implications for both model developers and policymakers. For instance, while the EU AI Act and California's SB-1047 aim to regulate AI models based on the number of floating point operations, we demonstrate that simple metrics like this alone fail to capture the full scope of AI's societal impact. We invite the community to explore and contribute to PersuasionArena and PersuasionBench, available at https://bit.ly/measure-persuasion, to advance our understanding of AI-driven persuasion and its societal implications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
