arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.13660cs.CLcs.AI
收藏

面向语言模型最小化定向更新的定向负训练方法研究

Towards Minimal Targeted Updates of Language Models with Targeted Negative Training

New York University(纽约大学);Google(谷歌)

Lily H. Zhang, Rajesh Ranganath, Arya Tafvizi

AI总结:

本研究针对生成式语言模型易产生非预期输出的问题,提出定向负训练(TNT)方法实现最小化定向更新,在减少不良输出的同时更好保留模型原有生成能力,为迭代约束式建模范式奠定基础。

摘要:

生成式语言模型展现出了令人瞩目的能力,但仍会将不可忽视的概率质量分配给不合要求的输出。在本研究中,我们致力于解决模型更新任务:在使模型避免产生非预期输出的同时,尽可能小幅度地改变模型其余行为,我们将这一挑战称为最小化定向更新。我们首先对最小化定向更新的概念进行了形式化定义,并提出了一种利用模型生成内容中的负样本实现这类更新的方法。我们提出的Targeted Negative Training(定向负训练,简称TNT)能够使更新后的分布与原始分布保持接近,这与现有处理负信号的损失函数不同——现有方法仅会压低概率,却无法控制更新后分布的具体形态。实验结果表明,与基线方法相比,TNT在减少非预期行为与维持模型生成能力之间实现了更优的权衡,为一种基于迭代训练更新的建模范式铺平了道路:该范式既能约束模型不生成不合要求的输出,又能保留模型已有的出色能力。

英文摘要:

Generative models of language exhibit impressive capabilities but still place non-negligible probability mass over undesirable outputs. In this work, we address the task of updating a model to avoid unwanted outputs while minimally changing model behavior otherwise, a challenge we refer to as a minimal targeted update. We first formalize the notion of a minimal targeted update and propose a method to achieve such updates using negative examples from a model's generations. Our proposed Targeted Negative Training (TNT) results in updates that keep the new distribution close to the original, unlike existing losses for negative signal which push down probability but do not control what the updated distribution will be. In experiments, we demonstrate that TNT yields a better trade-off between reducing unwanted behavior and maintaining model generation behavior than baselines, paving the way towards a modeling paradigm based on iterative training updates that constrain models from generating undesirable outputs while preserving their impressive capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑