Transformer-based language models are able to generate fluent text and be efficiently adapted across various natural language generation tasks. However, language models that are pretrained on large unlabeled web text corpora have been shown to suffer from degenerating toxic content and social bias behaviors, consequently hindering their safe deployment. Various detoxification methods were proposed to mitigate the language model's toxicity; however, these methods struggled to detoxify language models when conditioned on prompts that contain specific social identities related to gender, race, or religion. In this study, we propose Reinforce-Detoxify; A reinforcement learning-based method for mitigating toxicity in language models. We address the challenge of safety in language models and propose a new reward model that is able to detect toxic content and mitigate unintended bias towards social identities in toxicity prediction. The experiments demonstrate that the Reinforce-Detoxify method for language model detoxification outperforms existing detoxification approaches in automatic evaluation metrics, indicating the ability of our approach in language model detoxification and less prone to unintended bias toward social identities in generated content.
GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian University of Technology（大连理工大学）
Do Prompts Guarantee Safety? Mitigating Toxicity from LLM Generations through Subspace Intervention
Department of Computer Science（计算机科学系）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Department of Electronics and Communications Engineering（电子与通信工程系）
The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
Fudan University（复旦大学）；INFLY TECH（INFLY科技）；Griffith University（格里菲斯大学）；University of Waterloo（滑铁卢大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）
Quark: Controllable Text Generation with Reinforced Unlearning
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）
大规模的语言模型经常学习与用户期望不一致的行为。生成的文本可能包含攻击性或有毒语言，包含大量重复，或者与用户期望的不同情绪。我们考虑的任务是通过微调语言模型的信号不做什么来消除这些错位。我们引入了量化奖励约束（Quark），这是一种优化奖励函数的算法，该函数量化了（不）想要的属性，同时不会偏离原始模型太远。Quark在（i）收集具有当前语言模型的样本，（ii）基于奖励将它们分类为分位数，每个分位数由预先添加到语言模型的输入的奖励令牌标识，以及（iii）在来自每个分位数的以其奖励令牌为条件的样本上使用标准语言建模损失，同时通过KL发散惩罚保持在原始语言模型附近。通过在生成时以高奖励令牌为条件，该模型生成的文本显示出较少的不需要的属性。对于消除毒性，消极情绪和重复，我们的实验表明Quark优于强基线和最先进的强化学习方法，如PPO（Schulman et al. 2017），同时仅依赖于标准语言建模原语。
Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation
Nanjing University（南京大学）；Australia National University（澳大利亚国立大学）；Wuhan University（武汉大学）；National University of Singapore（新加坡国立大学）；University of Technology Sydney（技术科技大学）
Mitigating Overthinking in Large Reasoning Models via Difficulty-aware Reinforcement Learning
Laboratory for Artificial Intelligence and New Forms of Education（人工智能与新教育形式实验室）；National Engineering Research Center of Educational Big Data（教育大数据国家工程研究中心）；Faculty of Artificial Intelligence in Education, Central China Normal University（教育人工智能学院，中央财经大学）；College of Electronics and Information Engineering, South-Central Minzu University（电子与信息工程学院，西南民族大学）
Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
SKLCCSE, Beihang University（北航智能计算与复杂系统研究院）；National University of Singapore（新加坡国立大学）；Zhongguancun Laboratory, Beijing（中关村实验室）；Institute of Dataspace, Hefei（合肥数据空间研究院）；Nanyang Technological University（南洋理工大学）
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；New Jersey Institute of Technology（新泽西理工学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation
Institute for Logic, Language and Computation, University of Amsterdam（逻辑、语言与计算研究所，阿姆斯特丹大学）；Center for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）
