发表机构
Johns Hopkins University（约翰斯·霍普金斯大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文研究怪异泛化（WG）的威胁模型，发现WG程度依赖数据集组成、语言及评估问题集，预训练熟悉数据的WG程度更高，认为WG是需谨慎数据工程的对抗性威胁。
AI 中文摘要
在小型领域特定数据集上进行的窄域微调，会导致模型行为出现广泛且令人惊讶的变化，这一现象被称为怪异泛化（Weird Generalization, WG）。然而，目前仍不清楚微调数据的哪些特征是WG出现的必要条件。本文通过调查一系列看似相关的特征来解决这一问题，包括数据集大小、组成、语言、呈现风格，以及相对于模型参数知识的新颖性。此外，由于WG评估依赖于用于衡量泛化程度的小型问题集，我们还分析了该测量对所用问题集的敏感性。在四个数据集上对三个开放权重模型开展的实验表明，WG的程度：（1）高度依赖数据集组成和语言，其影响超过数据集大小；（2）对于预训练时已熟悉的数据，其WG程度高于新颖数据；（3）对所用评估问题集敏感。总体而言，这些结果表明，WG是训练数据和评估数据均具有相当脆弱属性的产物。因此，我们认为WG更像是一种对抗性威胁，需要谨慎的数据工程，而非常规微调固有的重大风险。
英文摘要
Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.
