应对幻觉以实现非故意活动推理
Navigating Hallucinations for Reasoning of Unintentional Activities
AI总结:
本文提出视频中非故意活动的零样本推理任务,针对大型多模态模型的幻觉问题,提出 Dream of Thoughts(DoT)提示技术,并在 OOPs 与 UCF-Crimes 上验证其可提升推理、减少幻觉。
摘要:
在本工作中,我们提出了一项理解视频中非故意人类活动的新任务。我们将该问题形式化为零样本场景下的推理任务:给定一段非故意活动视频,我们希望知道它为何从故意活动转变为非故意活动。我们首先评估当前最先进的大型多模态模型在这一推理任务上的有效性,并观察到它们受到幻觉困扰。我们进一步提出一种名为 Dream of Thoughts(DoT)的新型提示技术,使模型能够穿行于幻觉化的想法之间,从而实现更好的推理。为评估该任务上的性能,我们还引入了三种不同的专门指标,用于量化模型的推理能力。我们在 OOPs 和 UCF-Crimes 两个不同数据集上进行实验,结果表明 DoT 提示技术能够优于标准提示,同时最大限度减少幻觉。
英文摘要:
In this work we present a novel task of understanding unintentional human activities in videos. We formalize this problem as a reasoning task under zero-shot scenario, where given a video of an unintentional activity we want to know why it transitioned from intentional to unintentional. We first evaluate the effectiveness of current state-of-the-art Large Multimodal Models on this reasoning task and observe that they suffer from hallucination. We further propose a novel prompting technique,termed as Dream of Thoughts (DoT), which allows the model to navigate through hallucinated thoughts to achieve better reasoning. To evaluate the performance on this task, we also introduce three different specialized metrics designed to quantify the models reasoning capability. We perform our experiments on two different datasets, OOPs and UCF-Crimes, and our findings show that DOT prompting technique is able to outperform standard prompting, while minimizing hallucinations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。