超越文化意识的训练：大型语言模型中数据集语言结构的作用
The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models
AI总结：
本文研究了数据集语言结构对大型语言模型文化表现的影响，发现词汇导向的成分在不同模型中表现更稳定，而语义或多样性极端成分可能产生中性或有害效果。
摘要：
大型语言模型（LLMs）的全球部署引发了关于文化不一致性的担忧，然而用于文化适应的微调数据集的语言属性仍然了解甚少。我们采用以数据集为中心的文化对齐观点，并询问哪些微调数据集的语言属性与文化表现相关，这些属性在训练前是否具有预测性，以及这些影响在不同模型间如何变化。我们计算了阿拉伯语、中文和日语数据集的轻量级语言、语义和结构指标，并在每种语言中分别应用主成分分析。这种设计确保了所得到的成分捕捉了同一种语言数据集之间的变化，而非语言之间的差异。所得到的成分对应于广泛可解释的轴，涉及语义连贯性、表层词汇和句法多样性，以及词汇或结构丰富性，尽管其组成在不同语言中有所不同。我们对三种主要的LLM家族（LLaMA、Mistral、DeepSeek）进行了微调，并在文化知识、价值观和规范的基准上进行了评估。虽然PCA成分与下游性能相关，但这些关联强烈依赖于模型。通过受控子集干预，我们发现以词汇为导向的成分（PC3）是最稳健的，能够在不同模型和基准上产生更一致的表现，而强调语义或多样性极端（PC1-PC2）往往中性或有害。
英文摘要：
The global deployment of large language models (LLMs) has raised concerns about cultural misalignment, yet the linguistic properties of fine-tuning datasets used for cultural adaptation remain poorly understood. We adopt a dataset-centric view of cultural alignment and investigate which properties of post-training data are associated with cultural performance, whether they can guide data selection before fine-tuning, and how their effects vary across languages and model families. We compute lightweight linguistic, semantic, and structural metrics for Arabic, Chinese, and Japanese datasets and apply principal component analysis (PCA) separately within each language. The resulting components form broadly interpretable axes: PC1 is generally dominated by semantic structure, PC2 captures diversity and lexical variation, and PC3 reflects more language-specific organization. We fine-tune LLaMA, Mistral, and DeepSeek models and evaluate them on benchmarks of cultural knowledge, values, and norms. Although the PCA-derived dataset descriptors are associated with downstream performance, the strongest relationships vary across models, benchmarks, and languages, indicating that no single component serves as a universal predictor. Controlled, size-matched subset interventions further show that PCA-guided selection can improve cultural performance when the relevant component and direction are validated against random sampling. PC3 provides the strongest signal for Arabic, while High-PC1 is most effective for Japanese, particularly for LLaMA. Chinese results are weaker and more model-specific and remain exploratory because of smaller subset sizes. Overall, our findings show that lightweight dataset descriptors can support pre-training data diagnostics, but effective cultural adaptation requires language- and architecture-aware selection rather than a universal linguistic criterion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
