Measuring Misogyny in Natural Language Generation: Preliminary Results from a Case Study on two Reddit Communities
Australian Research Council Centre of Excellence for Automated Decision-Making and Society（澳大利亚研究委员会自动化决策与社会卓越中心）；Queensland University of Technology（昆士兰科技大学）；University of New South Wales（新南威尔士大学）
Aaron J. Snoswell, Lucinda Nelson, Hao Xue, Flora D. Salim, Nicolas Suzor, Jean Burgess
Generic `toxicity' classifiers continue to be used for evaluating the potential for harm in natural language generation, despite mounting evidence of their shortcomings. We consider the challenge of measuring misogyny in natural language generation, and argue that generic `toxicity' classifiers are inadequate for this task. We use data from two well-characterised `Incel' communities on Reddit that differ primarily in their degrees of misogyny to construct a pair of training corpora which we use to fine-tune two language models. We show that an open source `toxicity' classifier is unable to distinguish meaningfully between generations from these models. We contrast this with a misogyny-specific lexicon recently proposed by feminist subject-matter experts, demonstrating that, despite the limitations of simple lexicon-based approaches, this shows promise as a benchmark to evaluate language models for misogyny, and that it is sensitive enough to reveal the known differences in these Reddit communities. Our preliminary findings highlight the limitations of a generic approach to evaluating harms, and further emphasise the need for careful benchmark design and selection in natural language evaluation.
Machines in the Crowd? Measuring the Footprint of Machine-Generated Text on Reddit
DIMES Dept., University of Calabria Italy；IT University of Copenhagen\ Centre for AI Denmark；DIMES Dept., University of Calabria；IT University of Copenhagen\ Centre for AI
Critical Behavioral Traits Foster Peer Engagement in Online Mental Health Communities
Indraprastha Institute of Information Technology Delhi（德里英德拉普拉斯塔信息技术学院）；Empathy Rocks, Inc. dba mpathic.ai（Empathy Rocks公司（以mpathic.ai名义运营））；University of California Santa Barbara（加州大学圣巴巴拉分校）；University of Washington（华盛顿大学）；Indian Institute of Technology Delhi（印度理工学院德里分校）
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
Observatory on Social Media, Indiana University, USA（社交媒体观察站，印第安纳大学，美国）；Center Synergy of Systems, TUD Dresden University of Technology, Germany（系统协同中心，德累斯顿技术大学，德国）
Language, Place, and Social Media: Geographic Dialect Alignment in New Zealand
Computer Science and Software Engineering, University of Canterbury（坎特伯雷大学计算机科学与软件工程系）；Department of Linguistics, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校语言学系）；School of Psychology, Speech and Hearing, University of Canterbury（坎特伯雷大学心理学、语音与听力学校）；Department of Linguistics, University of Canterbury（坎特伯雷大学语言学系）
