SemEval-2024 共享任务 6：SHROOM，一个关于幻觉及相关可观察过度生成错误的共享任务
SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
AI总结：
本文报告 SemEval-2024 共享任务 SHROOM 的结果，该任务基于 4000 条多标注数据检测 NLG 系统流畅但不准确的幻觉输出，并分析了各参赛团队在微调与零样本提示策略下的表现。
摘要：
本文介绍了 SHROOM 的结果，这是一个专注于检测幻觉的共享任务：即自然语言生成（NLG）系统产生的输出虽然流畅但不准确。这类过度生成的情况危及许多 NLG 应用，在这些应用中正确性往往至关重要。该共享任务使用一个新构建的数据集进行，该数据集包含 4000 个模型输出，每个输出由 5 名标注者标注，涵盖 3 个 NLP 任务：机器翻译、释义生成和定义建模。该共享任务共由 58 位不同用户组成 42 支团队参与解决，其中 27 支团队选择撰写系统描述论文；他们共同在两个任务轨道上提交了超过 300 个预测集。我们观察到解决该任务方式中的若干关键趋势——许多参与者依赖少数几个模型，并且通常依赖合成数据进行微调或零样本提示策略。虽然大多数团队确实优于我们提出的基线系统，但得分最高系统的表现仍与对更具挑战性项目的随机处理相一致。
英文摘要：
This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling. The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 27 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled -- many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
