超越原子事实的长文本信息对齐评估
Long-Form Information Alignment Evaluation Beyond Atomic Facts
AI总结：
针对现有细粒度信息对齐评估忽略事实间依赖、易受蒙太奇式欺骗叙事攻击的问题，研究提出MontageLie基准与DoveScore框架，联合验证事实准确性与事件顺序一致性，性能提升超8%。
摘要：
信息对齐评估器对各类自然语言生成（NLG）评估任务及可信大语言模型（LLM）部署至关重要，可减少幻觉并提升用户信任度。当前如FactScore等细粒度方法单独验证事实，却忽略事实间的依赖关系，存在隐蔽的漏洞。本研究提出MontageLie这一具有挑战性的基准，它通过对真实陈述进行“蒙太奇”拼接构建欺骗性叙事，且不会引入显式幻觉。我们证明，无论是基于LLM的粗粒度评估器还是现有细粒度框架都易受这种攻击，AUC-ROC得分均低于65%。为实现更鲁棒的细粒度评估，我们提出DoveScore——一种联合验证事实准确性和事件顺序一致性的新型框架。通过建模事实间关系，DoveScore性能优于现有细粒度方法8%以上，为长文本对齐评估提供了更鲁棒的解决方案。我们的代码和数据集可在https://github.com/dannalily/DoveScore获取。
英文摘要：
Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
