arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.20545cs.AI
收藏

测量机器:评估生成式AI作为多元社会技术系统

Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems

The School of History and Philosophy of Science, Faculty of Science(历史与哲学科学学院,科学学院);The University of Sydney(悉尼大学)

Rebecca L. Johnson

AI总结:

本文提出MaSH循环框架,通过递归过程评估生成式AI作为多元社会技术系统,结合世界价值观调查数据和结构化提示,探讨价值观在交互中的生成与塑造。

摘要:

在测量理论中,仪器不仅记录现实,还帮助构成所观察的事物。生成式AI的评估同样如此:基准测试不仅测量,还塑造模型显现的样子。功能主义基准将模型视为孤立预测者,而规范性方法评估系统应为何。两者都掩盖了通过何种社会技术过程实施意义和价值观,风险在多元语境中重新现实化狭隘文化视角。本文提出描述性替代方案。它主张生成式AI必须作为多元社会技术系统评估,并发展出机器-社会-人类(MaSH)循环框架,用于追踪模型、用户和机构如何递归地共同构建意义和价值观。评估从评判输出转向考察价值观在交互中的生成。三项贡献包括:概念上,MaSH循环将评估重新定义为递归、具身过程;方法上,世界价值观基准引入基于世界价值观调查数据的分布方法,结构化提示集和锚点意识评分;实证上,通过早期GPT-3的价值漂移和房地产中的社会技术评估展示这些内容。最后章节基于参与现实主义,论证提示和评估是构成性干预,而非中性观察。本文主张静态基准不足以评估生成式AI。负责任的评估需要多元、过程导向的框架,使可见哪些价值观被实施。因此,评估是治理的场所,塑造AI系统被理解、部署和信任的方式。

英文摘要:

In measurement theory, instruments do not simply record reality; they help constitute what is observed. The same holds for generative AI evaluation: benchmarks do not just measure, they shape what models appear to be. Functionalist benchmarks treat models as isolated predictors, while prescriptive approaches assess what systems ought to be. Both obscure the sociotechnical processes through which meaning and values are enacted, risking the reification of narrow cultural perspectives in pluralist contexts. This thesis advances a descriptive alternative. It argues that generative AI must be evaluated as a pluralist sociotechnical system and develops Machine-Society-Human (MaSH) Loops, a framework for tracing how models, users, and institutions recursively co-construct meaning and values. Evaluation shifts from judging outputs to examining how values are enacted in interaction. Three contributions follow. Conceptually, MaSH Loops reframes evaluation as recursive, enactive process. Methodologically, the World Values Benchmark introduces a distributional approach grounded in World Values Survey data, structured prompt sets, and anchor-aware scoring. Empirically, the thesis demonstrates these through two cases: value drift in early GPT-3 and sociotechnical evaluation in real estate. A final chapter draws on participatory realism to argue that prompting and evaluation are constitutive interventions, not neutral observations. The thesis argues that static benchmarks are insufficient for generative AI. Responsible evaluation requires pluralist, process-oriented frameworks that make visible whose values are enacted. Evaluation is therefore a site of governance, shaping how AI systems are understood, deployed, and trusted.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑