arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.01767cs.CLcs.AIcs.CY
收藏

[Re]更坏的种子:再现词汇法测量偏差的评价

[Re] Badder Seeds: Reproducing the Evaluation of Lexical Methods for Bias Measurement

University of Amsterdam(阿姆斯特丹大学)

Jille van der Togt, Lea Tiyavorabun, Matteo Rosati, Giulio Starace

摘要:

在NLP中对抗偏见需要偏见测量。偏差测量几乎总是通过使用种子术语的词典来实现,即指定刻板印象或感兴趣的维度的单词集。这项再现性研究的重点是原作者的主要主张,即这些词典的构建原理需要在使用前进行彻底的检查,因为用于偏差测量的种子本身也会表现出偏差。本研究旨在评价论文中给出的定量和定性结果的重现性及其得出的结论。我们重现了大部分支持原作者一般主张的结果:种子集经常受到偏见的影响,影响其作为偏见指标基线的性能。一般来说,我们的结果反映了原始文件的。它们在某些情况下略有不同,但不会破坏论文显示种子集脆弱性的总体意图。

英文摘要:

Combating bias in NLP requires bias measurement. Bias measurement is almost always achieved by using lexicons of seed terms, i.e. sets of words specifying stereotypes or dimensions of interest. This reproducibility study focuses on the original authors' main claim that the rationale for the construction of these lexicons needs thorough checking before usage, as the seeds used for bias measurement can themselves exhibit biases. The study aims to evaluate the reproducibility of the quantitative and qualitative results presented in the paper and the conclusions drawn thereof. We reproduce most of the results supporting the original authors' general claim: seed sets often suffer from biases that affect their performance as a baseline for bias metrics. Generally, our results mirror the original paper's. They are slightly different on select occasions, but not in ways that undermine the paper's general intent to show the fragility of seed sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑