ML-EAT：一种用于可解释和透明社会科学的多层嵌入关联测试
ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science
AI总结：
本研究提出多层嵌入关联测试（ML-EAT），通过三个粒度层次量化偏见并定义九种EAT模式及可视化方法，提升了语言技术中内在偏见测量的可解释性与透明度。
摘要：
本研究引入了多层嵌入关联测试（ML-EAT），这是一种旨在以可解释且透明的方式测量语言技术中内在偏见的方法。ML-EAT通过在三个粒度递增的层次上量化偏见，解决了传统EAT测量中存在的模糊性和难以解释的问题：两个目标概念与两个属性概念之间的差异关联；每个目标概念与两个属性概念的个体效应量；以及每个单独目标概念与每个单独属性概念之间的关联。使用ML-EAT，本研究定义了一个EAT模式分类法，描述了嵌入关联测试的九种可能结果，每种结果都对应一个独特的EAT-Map，这是一种用于解释ML-EAT的新型四象限可视化方法。对静态和历时词嵌入、GPT-2语言模型以及CLIP语言与图像模型的实证分析表明，EAT模式增加了关于构成EAT的组件偏见的其他不可观察的信息；揭示了零样本模型中提示的效果；并且还能识别余弦相似度作为无效指标的情况，从而使EAT变得不可靠。我们的工作提供了一种使偏见更可观察和可解释的方法，提高了对人类心智和社会进行计算调查的透明度。
英文摘要：
This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in interpreting the traditional EAT measurement by quantifying bias at three levels of increasing granularity: the differential association between two target concepts with two attribute concepts; the individual effect size of each target concept with two attribute concepts; and the association between each individual target concept and each individual attribute concept. Using the ML-EAT, this research defines a taxonomy of EAT patterns describing the nine possible outcomes of an embedding association test, each of which is associated with a unique EAT-Map, a novel four-quadrant visualization for interpreting the ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2 language models, and a CLIP language-and-image model shows that EAT patterns add otherwise unobservable information about the component biases that make up an EAT; reveal the effects of prompting in zero-shot models; and can also identify situations when cosine similarity is an ineffective metric, rendering an EAT unreliable. Our work contributes a method for rendering bias more observable and interpretable, improving the transparency of computational investigations into human minds and societies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
