校准给谁看？JEV中人物角色与语言对文化价值观的影响

中文摘要 AI 辅助

仅做决策的语言模型为每个答案选项返回一个概率，而非生成文本，这使得它们作为调查受访者和评判者具有吸引力。我们使用价值观调查模块2013（Values Survey Module 2013）审计了TypeSafe公司的JEV模型的文化价值观。我们以12个匹配的沙特人物角色、12个匹配的美国人物角色以及无人物角色，在英语和阿拉伯语下，以八种请求表述方式询问了24个条目（共288,000个答案）。JEV的答案高度可重复（组内相关系数ICC为0.997），且在无人物角色时，其答案与其自身美国人物角色的答案相似。当人物角色为沙特而非美国时，答案朝着人类沙特与美国差异的方向移动，在英语中重现了该差异大小的87%，但在阿拉伯语中仅为62%，且长期导向出现反转。一项语言交叉实验表明，阿拉伯语中较小的差异来自条目语言，而非人物角色描述的语言。年龄对画像的偏移程度与国籍相当，性别对沙特人物角色的偏移程度大于对美国人物角色的偏移，且JEV在阿拉伯语和沙特人物角色下自信度较低。这些模式在所有请求设计中均保持一致，尽管该模型从不生成文本。

英文摘要

Decision-only language models return a probability for every answer option instead of generating text, which makes them attractive as survey respondents and as judges. We audit the cultural values of one such model, TypeSafe's JEV, with the Values Survey Module 2013. We asked it the 24 items as 12 matched Saudi and 12 matched American personas and without a persona, in English and Arabic, under eight ways of formulating the request (288,000 answers). JEV's answers were highly repeatable (ICC 0.997), and without a persona they resembled those of its own American personas. When the persona was Saudi rather than American, the answers moved in the direction of the human Saudi-US difference, reproducing 87% of its size in English but 62% in Arabic, with long-term orientation reversed. A language cross shows that the smaller difference in Arabic comes from the language of the items, not from the language of the persona description. Age shifted the profiles about as much as nationality, gender shifted them more for Saudi than for American personas, and JEV was less confident in Arabic and for Saudi personas. These patterns held in every request design, although the model never generates text.

发表机构

• Alfaisal University（阿尔费萨尔大学）

机构由 AI 辅助整理，请以论文原文为准。