“推理”与修辞：LLM生成的反驳论证中风格与证据的权衡
"Reasoning" with Rhetoric: On the Style-Evidence Tradeoff in LLM-Generated Counter-Arguments
AI总结：
本研究评估LLM生成风格化反驳论证的有效性，发现人类偏好风格化输出，但模型仍存在风格与证据的权衡，未达人类标准，并探讨了伦理风险。
摘要：
大型语言模型（LLMs）在生成基于证据且具有风格的反驳论证中发挥着关键作用，但其在实际应用中的有效性尚未得到充分探索。以往研究往往忽视了证据性和风格之间的平衡，而这两者对于有说服力的论证至关重要。为解决这一问题，我们评估了在Counterfire数据集中风格化基于证据的反驳论证生成的有效性，该数据集包含38,000个反驳论证，通过修改Reddit的ChangeMyView社区的反驳论证以遵循不同的论述风格而生成。我们评估了来自基础模型和微调模型（如GPT-3.5、PaLM-2和Koala-13B）以及较新模型（GPT-4o、Claude Haiku、LLaMA-3.1）的通用和风格化反驳论证，重点关注修辞质量和说服力。我们的研究发现，人类更偏好风格化的反驳论证而非原始输出，其中GPT-3.5 Turbo表现良好，但仍未达到人类在修辞质量和说服力方面的标准，表明LLM在生成反驳论证时仍存在持久的风格与证据权衡。最后，我们探讨了LLM说服研究中的伦理考量，涉及欺骗性实践的潜在风险以及透明部署方法的必要性，以防止在公共话语中被滥用。代码和数据集可在https://github.com/Preetika764/Style_control/获取。
英文摘要：
Large language models (LLMs) play a key role in generating evidence-based and stylistic counter-arguments, yet their effectiveness in real-world applications has been underexplored. Previous research often neglects the balance between evidentiality and style, which are crucial for persuasive arguments. To address this, we evaluated the effectiveness of stylized evidence-based counter-argument generation in Counterfire, a new dataset of 38,000 counter-arguments generated by revising counter-arguments to Reddit's ChangeMyView community to follow different discursive styles. We evaluated generic and stylized counter-arguments from basic and fine-tuned models such as GPT-3.5, PaLM-2, and Koala-13B, as well as newer models (GPT-4o, Claude Haiku, LLaMA-3.1) focusing on rhetorical quality and persuasiveness. Our findings reveals that humans prefer stylized counter-arguments over the original outputs, with GPT-3.5 Turbo performing well, though still not reaching human standards of rhetorical quality nor persuasiveness indicating a persisting style-evidence tradeoff in counter-argument generation by LLMs. We conclude with an examination of ethical considerations in LLM persuasion research, addressing potential risks of deceptive practices and the need for transparent deployment methodologies to safeguard against misuse in public discourse. The code and dataset are available at https://github.com/Preetika764/Style_control/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
