arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.19650cs.CLcs.AI

HausaNLP在SemEval-2025任务3：迈向细粒度模型感知的幻觉检测

HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection

HausaNLP（豪萨语自然语言处理研究组）；University of Abuja（阿布贾大学）；Bayero University Kano（卡诺贝罗大学）；University of Pretoria（比勒陀利亚大学）；Imperial College London（伦敦帝国学院）；Northeastern University（东北大学）
Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muha… 展开作者

Maryam Bala, Amina Imam Abubakar, Abdulhamid Abubakar, Abdulkadir Shehu Bichi, Hafsa Kabir Ahmad, Sani Abdullahi Sani, Idris Abdulmumin, Shamsuddeen Hassan Muhamad, Ibrahim Said Ahmad

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出使用自然语言推理和微调ModernBERT模型，在MU-SHROOM共享任务中检测英语幻觉，取得IoU 0.032和相关性0.422，揭示了幻觉检测的复杂性。

摘要：

本文介绍了我们在多语言幻觉及相关可观察过度生成错误共享任务（MU-SHROOM）中的研究发现，该任务聚焦于识别大型语言模型（LLMs）中的幻觉及相关过度生成错误。该共享任务涉及检测LLMs在14种语言中生成输出中构成幻觉的特定文本片段。为了解决这一任务，我们旨在提供对英语中幻觉发生及其严重性的细致、模型感知的理解。我们使用了自然语言推理，并利用一个包含400个样本的合成数据集对ModernBERT模型进行了微调，取得了0.032的交并比（IoU）分数和0.422的相关性分数。这些结果表明，模型的置信度分数与幻觉的实际存在之间存在中等程度的正相关。IoU分数表明，我们的模型在预测的幻觉片段与真实标注之间的重叠度相对较低。鉴于幻觉检测的复杂性，这一性能并不令人意外。幻觉通常以微妙的方式表现，依赖于上下文，使得精确定位其边界变得困难。

英文摘要：

This paper presents our findings of the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes, MU-SHROOM, which focuses on identifying hallucinations and related overgeneration errors in large language models (LLMs). The shared task involves detecting specific text spans that constitute hallucinations in the outputs generated by LLMs in 14 languages. To address this task, we aim to provide a nuanced, model-aware understanding of hallucination occurrences and severity in English. We used natural language inference and fine-tuned a ModernBERT model using a synthetic dataset of 400 samples, achieving an Intersection over Union (IoU) score of 0.032 and a correlation score of 0.422. These results indicate a moderately positive correlation between the model's confidence scores and the actual presence of hallucinations. The IoU score indicates that our model has a relatively low overlap between the predicted hallucination span and the truth annotation. The performance is unsurprising, given the intricate nature of hallucination detection. Hallucinations often manifest subtly, relying on context, making pinpointing their exact boundaries formidable.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑