arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.05917cs.IRstat.AP
收藏

审计推荐系统中人工标注标签的跨文化一致性

Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems

Rock Yuren Pang, Jack Cenatempo, Franklyn Graham, Bridgette Kuehn, Maddy Whisenant, Portia Botchway, Katie Stone Perez, Allison Koenecke

AI总结:

本研究通过调查5,174名游戏玩家,审计了推荐系统中人工标注标签的跨文化一致性,发现文化语言差异导致概念化不一致,并证明全球训练集优于单一国家训练集,提出了可推广的审计框架。

摘要:

推荐系统越来越依赖于大规模的人工标注数据集;然而,受雇生成这些标签的人工标注者越来越多地来自同质化背景。当基于这些标签的下游预测模型被全球性地应用于异构用户群体时,这就产生了一个问题。我们针对标签本身研究这种脱节,探讨不同人口统计特征的标注者之间对标签的“概念化是否一致”。在一项关于视频游戏标签的案例研究中,我们对5,174名游戏玩家进行了调查,识别出一组概念化不一致的游戏标签,进行了因果分析,并提出了导致跨国标签标注差异的文化和语言原因。我们进一步证明,与同质化(单一国家)训练集相比,游戏标注的预测模型在全球训练集上表现更好。最后,我们为从业者提供了一个可推广的框架,用于审计他们自己的数据标注过程以确保标签概念化的一致性,并鼓励从业者从标注者招募和数据标注的早期阶段就开始考虑推荐系统中的全球包容性。

英文摘要:

Recommendation systems increasingly depend on massive human-labeled datasets; however, the human annotators hired to generate these labels increasingly come from homogeneous backgrounds. This poses an issue when downstream predictive models -- based on these labels -- are applied globally to a heterogeneous set of users. We study this disconnect with respect to the labels themselves, asking whether they are ``consistently conceptualized'' across annotators of different demographics. In a case study of video game labels, we conduct a survey on 5,174 gamers, identify a subset of inconsistently conceptualized game labels, perform causal analyses, and suggest both cultural and linguistic reasons for cross-country differences in label annotation. We further demonstrate that predictive models of game annotations perform better on global train sets as opposed to homogeneous (single-country) train sets. Finally, we provide a generalizable framework for practitioners to audit their own data annotation processes for consistent label conceptualization, and encourage practitioners to consider global inclusivity in recommendation systems starting from the early stages of annotator recruitment and data-labeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑