SceneAlign: 在复杂视觉场景中将多模态推理对齐到场景图
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes
AI总结：
SceneAlign通过利用场景图进行结构干预，提升多模态推理在复杂视觉场景中的准确性和忠实性。
摘要：
多模态大语言模型在复杂视觉场景中进行忠实推理时常常遇到困难，其中复杂的实体和关系需要在每一步都进行精确的视觉定位。这种推理不忠实性通常表现为幻觉实体、关系定位错误、步骤跳过和过度指定的推理。现有的基于偏好的方法通常依赖于文本扰动或答案条件化的理由，无法解决这一挑战，因为它们允许模型利用语言先验来绕过视觉定位。为了解决这一问题，我们提出了SceneAlign，一个利用场景图作为结构化视觉信息进行可控结构干预的框架。通过识别推理关键节点，并通过四种针对性策略扰动它们，以模仿典型的定位失败，SceneAlign构建了硬负理由，这些理由在语言上是合理的，但基于不准确的视觉事实。这些对比对在直接偏好优化中被用来引导模型朝着细粒度、结构忠实的推理发展。在七个视觉推理基准测试中，SceneAlign一致地提高了回答准确性和推理忠实性，突显了对视觉定位的意识对多模态推理的有效性。
英文摘要：
Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful reasoning. Across seven visual reasoning benchmarks, SceneAlign consistently improves answer accuracy and reasoning faithfulness, highlighting the effectiveness of grounding-aware alignment for multimodal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
