In measurement theory, instruments do not simply record reality; they help constitute what is observed. The same holds for generative AI evaluation: benchmarks do not just measure, they shape what models appear to be. Functionalist benchmarks treat models as isolated predictors, while prescriptive approaches assess what systems ought to be. Both obscure the sociotechnical processes through which meaning and values are enacted, risking the reification of narrow cultural perspectives in pluralist contexts.
This thesis advances a descriptive alternative. It argues that generative AI must be evaluated as a pluralist sociotechnical system and develops Machine-Society-Human (MaSH) Loops, a framework for tracing how models, users, and institutions recursively co-construct meaning and values. Evaluation shifts from judging outputs to examining how values are enacted in interaction.
Three contributions follow. Conceptually, MaSH Loops reframes evaluation as recursive, enactive process. Methodologically, the World Values Benchmark introduces a distributional approach grounded in World Values Survey data, structured prompt sets, and anchor-aware scoring. Empirically, the thesis demonstrates these through two cases: value drift in early GPT-3 and sociotechnical evaluation in real estate. A final chapter draws on participatory realism to argue that prompting and evaluation are constitutive interventions, not neutral observations.
The thesis argues that static benchmarks are insufficient for generative AI. Responsible evaluation requires pluralist, process-oriented frameworks that make visible whose values are enacted. Evaluation is therefore a site of governance, shaping how AI systems are understood, deployed, and trusted.
Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
Microsoft Research Asia(微软亚洲研究院);Fudan University(复旦大学);Tsinghua University(清华大学);The University of Chicago(芝加哥大学);Renmin University of China(中国人民大学);William & Mary(威廉与玛丽大学)
ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
Artificial Intelligence and Intelligent Information Systems, University of Trier(人工智能与智能信息系统,特里尔大学);German Research Center for Artificial Intelligence (DFKI), Trier(德国人工智能研究中心(DFKI),特里尔)