评估用于反犹事件分类的大语言模型
Evaluating Large Language Models for Antisemitic Incident Classification
AI总结：
研究利用大语言模型对反犹事件报告进行细粒度分类，评估了OpenAI的GPT-4o和Meta的Llama-3.2-3B-Instruct，发现其有潜力但需改进，提示中提供定义和示例可提升性能，还通过案例展示其能助力发现现实事件。
摘要：
应对社会中的仇恨和暴力需要从公共报告中及时检测仇恨事件，但自动识别仇恨事件仍未得到充分探索。我们引入仇恨事件检测任务，研究人工智能系统，特别是大语言模型（LLMs）发现和分类反犹事件报告并进行细粒度标注的能力。我们在多个包含来自新闻文章、民间社会报告和官方记录的反犹事件描述的专家标注数据集上评估了OpenAI的GPT-4o和Meta的Llama-3.2-3B-Instruct。我们表明，大语言模型，特别是GPT-4o，在这项任务中有潜力，但仍需要大幅改进。在提示中提供明确的术语定义和上下文示例可以提高性能：定义对以言辞为导向的事件（如经典的反犹比喻）最有帮助，而示例有助于标注以行动为导向的事件（如人身攻击）。一项对大学报纸的案例研究表明，大语言模型可以帮助发现相关的现实世界事件，支持早期监测和干预。总体而言，我们的研究结果突出了人工智能识别复杂危害能力方面的机遇和关键差距，并强调了人工智能开发者、政策制定者和民间社会之间需要共同努力，以设计模型、实施稳健的评估，并制定政策框架，以高效有效地定义和打击仇恨。
英文摘要：
Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
