大型语言模型中的文化对齐：基于软提示调优
Cultural Alignment in Large Language Models Using Soft Prompt Tuning
AI总结：
针对LLM文化对齐中缺乏逐token梯度的问题，提出用差分进化优化软提示编码文化行为，冻结权重且无需偏好数据，实验表明能减少与VSM13差异并提升与WVS的一致性。
摘要：
大型语言模型（LLM）的对齐通常通过监督微调或强化学习实现，这两种方法都需要标注数据或偏好数据，并更新模型权重。然而，若缺乏针对性的文化适应，已部署的LLM往往表现出文化同质化行为，无法反映多样化的地方价值观。将模型对齐到文化价值框架（如Hofstede的价值调查模块VSM13）面临独特挑战：对齐信号仅以聚合的问卷级分数形式提供，这些分数是在模型生成对整个问卷的响应后计算得出的，因此不提供逐token的梯度，且按构造无需偏好数据。这使得标准的基于梯度的方法不适合此任务。我们提出一种部署友好的方法，将文化行为编码为短的、可调优的软提示，并使用差分进化（DE）进行优化，同时保持模型权重冻结且无需偏好数据。在推理时，系统插入相应的文化特定提示，以适应不同文化的模型响应。在四个国家和四个指令调优模型上的实验表明，DE优化的提示通常能减少与VSM13参考配置文件的差异，提高与世界价值观调查（WVS）的排名一致性（WVS是优化过程中未见过的独立框架），并且在三个LLM评委的多数投票盲法成对评估中更受青睐。
英文摘要：
Large Language Model (LLM) alignment is commonly achieved through supervised fine-tuning or reinforcement learning, both of which require labeled or preference data and update model weights. Without targeted cultural adaptation, however, deployed LLMs often exhibit culturally homogeneous behavior that fails to reflect diverse local values. Aligning models to cultural value frameworks such as Hofstede's Value Survey Module (VSM13) presents a distinct challenge: alignment signals are available only as aggregated survey-level scores computed after generating responses to an entire survey, providing no per-token gradient and requiring no preference data by construction. This makes standard gradient-based alignment methods ill-suited to the task. We propose a deployment-friendly approach that encodes cultural behavior in short, tunable soft prompts optimized with Differential Evolution (DE), while keeping model weights frozen and requiring no preference data. At inference, the system inserts the appropriate cultural-specific prompt to adapt model responses for different cultures. Experiments across four countries and four instruction-tuned models show that DE-optimized prompts generally reduce discrepancy with VSM13 reference profiles, improve rank agreement with the World Values Survey (WVS), an independent framework not seen during optimization, and are preferred in blinded pairwise evaluations using majority voting across three LLM judges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
