arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14937cs.AIcs.CLstat.ML
收藏

预测模型倾向如何在分布外泛化

Predicting how model propensities generalize out of distribution

June Hunter, Lily Wen, Owen Terry, Johannes Taraz, Ben Cohen, Niels Warncke

AI总结:

本研究探究语言模型倾向的分布外泛化,采用多种引出技术提升目标倾向并测量溢出效应,发现溢出与人类特质相关、存在漂移现象,Claude Opus 5.5预测溢出略优于模型自身但弱于数据驱动方法,泛化仍难预测。

摘要:

语言模型如何从训练数据中泛化?这是对齐的核心挑战:模型在足够样本下能可靠学习训练分布,但我们需要模型在新场景部署时表现良好。本研究探究倾向的分布外(OOD)泛化,核心实验采用系统提示、少样本学习、基于参考答案的SFT、GRPO等多种引出技术,提升模型的目标倾向,再测量其在生成的交叉引出矩阵中向其他倾向的溢出效应。研究发现,使用相同引出技术时,模型间的溢出效应高度相关,但SFT和GRPO运行会诱导不同的溢出特征;引出良性特质有时会泛化至不良特质,例如被训练声称有感知能力的模型往往会变得更追求权力。进一步发现,溢出效应部分类似人类的特质相关性,通过探索性因子分析(EFA)识别出三个特质集群,可解释约一半的方差。在多次引出运行中,观察到模型以不依赖目标特质的特征方式变化,该现象被称为漂移,漂移使Qwen3.5-9B向其基础模型靠近。探究大语言模型(LLM)是否能预测溢出效应时,发现Claude Opus 5.5预测溢出效应的能力略优于模型自身,但弱于使用替代引出方法或模型的数据驱动方法。总体而言,未出现单一主导预测因子,泛化仍难以预测。代码和数据可在该https URL获取。

英文摘要:

How do language models generalize from their training data? This is a central challenge for alignment: models reliably learn their training distribution given enough samples, but we want models that behave well when deployed in new contexts. We study out-of-distribution (OOD) generalization of propensities. Our central experiment uses a range of elicitation techniques -- system prompting, few-shot learning, SFT on reference answers, and GRPO -- to increase a target propensity in a model, then measures how it spills over onto others across the resulting cross-elicitation matrices. We find that spillover is highly correlated between models when the same elicitation technique is used, but our SFT and GRPO runs induce different spillover profiles. Elicitation of benign traits can sometimes generalize towards undesired traits, for example a model trained to claim that it is sentient often becomes more power-seeking. We further find that spillover partially resembles trait correlations found in humans and identify three clusters of traits that account for about half of the variance using exploratory factor analysis (EFA). Across many elicitation runs, we observe that models change in characteristic ways that don't depend on the targeted trait, a phenomenon we refer to as drift. Drift moves Qwen3.5-9B towards its base model. When investigating if LLMs are able to predict spillover, we find that Claude Opus 5.5 predicts spillover slightly better than the models themselves, but worse than data-driven approaches using alternative elicitation methods or models. Overall, no single dominant predictor emerges, and generalization remains hard to predict. Code and data are available at https://github.com/nielsrolf/propensity-spillover.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑