arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.00418cs.LGcs.AI
收藏

泛化信任:语言模型中的弱到强可信度

Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models

Harvard University(哈佛大学)

Martin Pawelczyk, Lillian Sun, Zhenting Qi, Aounon Kumar, Himabindu Lakkaraju

AI总结:

本文首次研究语言模型中可信度属性能否通过弱到强泛化迁移,提出Weak TFT和Weak+WTS TFT两种训练策略,发现公平性与鲁棒性可迁移而隐私难以迁移。

摘要:

生成式人工智能,尤其是大语言模型的迅速普及,已使其融入各类应用之中。一个被称为弱到强泛化的关键现象——即在弱模型输出上训练的强模型在任务性能上超越弱模型——已引起广泛关注。然而,诸如鲁棒性、公平性和隐私等关键可信度属性能否以类似方式泛化,仍是一个悬而未决的问题。在本工作中,我们通过考察一个更强的模型在弱模型输出上微调时能否继承可信度属性来研究这一问题,我们将这一过程称为弱到强可信度泛化。为此,我们引入了两种基础训练策略:1)弱可信度微调(Weak TFT),在弱模型微调过程中利用可信度正则化;2)弱及弱到强可信度微调(Weak+WTS TFT),将正则化扩展到弱模型和强模型。我们在真实世界数据集上的实验评估表明,虽然部分可信度属性(如公平性、对抗鲁棒性和分布外鲁棒性)在两个模型都经过正则化时迁移效果显著提升,但隐私等其他属性并未表现出弱到强可信度的迹象。作为首个通过弱到强泛化探索可信度泛化的研究,我们的工作为弱到强泛化的潜力与局限性提供了有价值的见解。

英文摘要:

The rapid proliferation of generative AI, especially large language models, has led to their integration into a variety of applications. A key phenomenon known as weak-to-strong generalization - where a strong model trained on a weak model's outputs surpasses the weak model in task performance - has gained significant attention. Yet, whether critical trustworthiness properties such as robustness, fairness, and privacy can generalize similarly remains an open question. In this work, we study this question by examining if a stronger model can inherit trustworthiness properties when fine-tuned on a weaker model's outputs, a process we term weak-to-strong trustworthiness generalization. To address this, we introduce two foundational training strategies: 1) Weak Trustworthiness Finetuning (Weak TFT), which leverages trustworthiness regularization during the fine-tuning of the weak model, and 2) Weak and Weak-to-Strong Trustworthiness Finetuning (Weak+WTS TFT), which extends regularization to both weak and strong models. Our experimental evaluation on real-world datasets reveals that while some trustworthiness properties, such as fairness, adversarial, and OOD robustness, show significant improvement in transfer when both models were regularized, others like privacy do not exhibit signs of weak-to-strong trustworthiness. As the first study to explore trustworthiness generalization via weak-to-strong generalization, our work provides valuable insights into the potential and limitations of weak-to-strong generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑