WinoPron:重新审视英语Winogender Schemas以提升一致性、覆盖度与语法格规范
WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case
AI总结:
本研究针对Winogender Schemas数据集的现有缺陷进行修正,推出新数据集WinoPron,通过评估多类主流共指消解模型证实宾格代词消解难度更高,还提出非二元的代词偏差评估方法,揭示偏差在代词不同表层形式间的差异。
摘要:
尽管衡量共指消解中的偏差与鲁棒性是重要目标,但这类测量的有效性完全依赖于我们所用的测量工具。Winogender Schemas(Rudinger等人,2018)是一个颇具影响力的数据集,旨在评估共指消解中的性别偏差,但深入审视后发现该数据存在若干问题,会影响其可靠评估的效用,包括将不同代词形式视为等价、违反模板约束以及排版错误。我们识别出这些问题并进行了修正,推出了新数据集WinoPron。借助WinoPron,我们评估了两个最先进的监督式共指消解系统——SpanBERT以及五种规模的FLAN-T5——并证实宾格代词对所有模型而言都更难消解。我们还提出了一种评估共指消解中代词偏差的新方法,该方法突破了二元评估的局限。通过这一方法我们还发现,偏差特征不仅在不同代词集合(如he与she)间存在差异,在这些集合的不同表层形式(如him与his)之间也存在差异。
英文摘要:
While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them. Winogender Schemas (Rudinger et al., 2018) are an influential dataset proposed to evaluate gender bias in coreference resolution, but a closer look reveals issues with the data that compromise its use for reliable evaluation, including treating different pronominal forms as equivalent, violations of template constraints, and typographical errors. We identify these issues and fix them, contributing a new dataset: WinoPron. Using WinoPron, we evaluate two state-of-the-art supervised coreference resolution systems, SpanBERT, and five sizes of FLAN-T5, and demonstrate that accusative pronouns are harder to resolve for all models. We also propose a new method to evaluate pronominal bias in coreference resolution that goes beyond the binary. With this method, we also show that bias characteristics vary not just across pronoun sets (e.g., he vs. she), but also across surface forms of those sets (e.g., him vs. his).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。