HalluWorld: 一个用于通过参考世界模型控制幻觉的基准
HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
AI总结：
本文提出HalluWorld基准，通过显式参考世界模型研究语言模型的幻觉问题，发现不同任务中幻觉表现不一致，表明幻觉源于多种失败模式而非单一能力。
摘要：
幻觉仍然是大语言模型的核心失败模式，但现有基准在摘要、问答、检索增强生成和代理交互中操作不一致。这种碎片化使得不清楚一种缓解措施在不同情境中是否有效。当前基准要么需要人工标注和固定参考，要么依赖难以复现的观察。为研究根本原因，我们引入HalluWorld，一个基于显式参考世界模型的可扩展基准：当模型生成一个与该世界不一致的可观察声明时，即产生幻觉。基于这一观点，我们构建了合成和半合成环境，在其中参考世界完全指定，模型观点受控，幻觉标签自动产生。HalluWorld涵盖网格世界、国际象棋和现实终端任务，使世界复杂性、可观察性、时间变化和源冲突政策可控，并将幻觉细分为细粒度错误类别。我们评估了前沿和开放权重语言模型在这些设置中的表现，发现一致模式：前沿模型在直接观察信息上的感知幻觉接近解决，而多步状态跟踪和因果正向模拟仍然困难且未被扩展思考普遍解决。在终端设置中，模型在何时应放弃时也遇到困难。不同探测类型和领域中的失败分布不均，表明幻觉源于不同的失败模式而非单一能力。我们的结果表明，受控参考世界为测量和减少现代语言模型中的幻觉提供了可扩展且可重复的路径。
英文摘要：
Hallucination remains a central failure mode of large language models, but existing benchmarks operationalize it inconsistently across summarization, question answering, retrieval-augmented generation, and agentic interaction. This fragmentation makes it unclear whether a mitigation that works in one setting reduces hallucinations across contexts. Current benchmarks either require human annotation and fixed references that may be memorized, or rely on observations in settings that are difficult to reproduce. To study root causes, we introduce HalluWorld, an extensible benchmark grounded in an explicit reference-world formulation: a model hallucinates when it produces an observable claim that is false with respect to this world. Building on this view, we construct synthetic and semi-synthetic environments in which the reference world is fully specified, the model's view is controlled, and hallucination labels are generated automatically. HalluWorld spans gridworlds, chess, and realistic terminal tasks, enabling controlled variation of world complexity, observability, temporal change, and source-conflict policy, and disentangling hallucinations into fine-grained error categories. We evaluate frontier and open-weight language models across these settings and find consistent patterns: perceptual hallucination on directly observed information is near-solved for frontier models, while multi-step state tracking and causal forward simulation remain difficult and are not generally solved by extended thinking. In the terminal setting, models also struggle with when to abstain. The uneven profile of failures across probe types and domains suggests that hallucinations arise from distinct failure modes rather than a single capability. Our results suggest that controlled reference worlds offer a scalable and reproducible path toward measuring and reducing hallucinations in modern language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
