大型语言模型在噪声证据下的推理能力如何？一个主动视觉推理基准

中文摘要 AI 辅助

现实世界中的推理很少简化为静态问答：智能体必须主动从工具和传感器中收集信息，而这些工具和传感器往往带有噪声且不可靠。然而，大多数现有的主动推理基准假设环境反馈是可信的，或者引入噪声却不提供明确的、校准过的不确定性信号，从而使得当证据本身不确定时，大型语言模型应如何推理的问题悬而未决。我们引入了VisualNoiseQA，这是一个在噪声视觉反馈下进行主动推理的新基准。一个纯文本的大型语言模型必须通过迭代查询一个固定的、现成的视觉语言模型（该模型被视为随机视觉传感器）来解决视觉问答问题。对于每次查询，我们抽取多个样本，并通过自一致性暴露经验不确定性信号，使推理者能够从不同角度探查，并决定下一步询问什么以及何时停止。我们的构建是自动且可扩展的：从多样化的视觉问答来源和两个带噪声的视觉语言模型出发，我们仅保留那些传感器不一致但人类可解答的问题。我们在涵盖感知、图表理解和知识密集型推理的1000个实例上评估了多个大型语言模型推理者。因此，VisualNoiseQA提供了一个受控的实验场，用于研究不同大型语言模型如何利用不确定性信号进行稳健推理。

英文摘要

Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy and unreliable. Yet most existing active reasoning benchmarks assume that environmental feedback is trustworthy, or introduce noise without exposing an explicit, calibrated uncertainty signal, leaving open how LLMs should reason when the evidence itself is uncertain. We introduce VisualNoiseQA, a novel benchmark for active reasoning under noisy visual feedback. A text-only LLM must solve VQA problems by iteratively querying a fixed, off-the-shelf VLM treated as a stochastic visual sensor. For each query, we draw multiple samples and expose an empirical uncertainty signal via self-consistency, enabling the reasoner to probe from different angles and decide what to ask next and when to stop. Our construction is automatic and scalable: starting from diverse VQA sources and two noisy VLMs, we retain only questions where the sensor is inconsistent yet human-solvable. We evaluate multiple LLM reasoners on 1,000 instances spanning perception, chart understanding, and knowledge-intensive reasoning. VisualNoiseQA thus provides a controlled playground to study how different LLMs exploit uncertainty signals for robust reasoning.

发表机构

• Arizona State University（亚利桑那州立大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

27 pages, 9 figures, 11 tables