基于LLM的机器反馈强化学习不当论点重写
LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback
AI总结：
本文提出一种基于强化学习的LLM重写方法，通过平衡内容保留与适当性来缓解论点中的不当语言，在文档级别进行改写，实验证明其显著优于少样本学习、提示和人类等基线方法。
摘要：
确保在线讨论的文明和富有成效是社交媒体平台面临的一项重大挑战。此类平台通常依赖用户和自动检测工具来标记其他用户的不当论点，然后由审核员进行审查。然而，这种事后审核既昂贵又耗时，审核员常常被标记内容的数量和严重性所淹没。相反，一个有前景的替代方案是在内容创建期间防止负面行为。本文研究如何通过计算方式缓解论点中的不当语言。我们提出了一种基于强化学习的重写方法，该方法基于现有分类器平衡内容保留和适当性，提示指令微调大型语言模型（LLM）作为我们的初始策略。与相关的风格转换任务不同，重写不当论点允许永久删除和添加内容。因此，它在文档级别而非句子级别进行处理。我们在绝对和相对人类评估研究中评估了奖励函数的不同加权方案。在非平行数据上的系统实验证明，我们的方法能够在很大程度上保留论点内容的同时缓解其不当性。它显著优于包括少样本学习、提示和人类在内的竞争基线。
英文摘要：
Ensuring that online discussions are civil and productive is a major challenge for social media platforms. Such platforms usually rely both on users and on automated detection tools to flag inappropriate arguments of other users, which moderators then review. However, this kind of post-hoc moderation is expensive and time-consuming, and moderators are often overwhelmed by the amount and severity of flagged content. Instead, a promising alternative is to prevent negative behavior during content creation. This paper studies how inappropriate language in arguments can be computationally mitigated. We propose a reinforcement learning-based rewriting approach that balances content preservation and appropriateness based on existing classifiers, prompting an instruction-finetuned large language model (LLM) as our initial policy. Unlike related style transfer tasks, rewriting inappropriate arguments allows deleting and adding content permanently. It is therefore tackled on document level rather than sentence level. We evaluate different weighting schemes for the reward function in both absolute and relative human assessment studies. Systematic experiments on non-parallel data provide evidence that our approach can mitigate the inappropriateness of arguments while largely preserving their content. It significantly outperforms competitive baselines, including few-shot learning, prompting, and humans.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
